Dude, sounds like your hull is dragging. That's a weight transfer issue.
Be prepared to shift your weight in the boat as you're changing your boat's state.
From standstill your crew will have to move their weight from the stern to the bow as you build speed to get you planing and then come back a bit once you're there.
It's an organic thing.
But from the off you have to make sure that the standards of your boat are just that.
Let your SIB assume the temp of the water and get it right once the boat has met the water temp, so take the pump with you and fill once you've been on the water for about ten minutes. It's not just the tubes that meet the temp but the air inside them, and generally they will cool and make your boat bendy.
Imagine the weight transfer a motocross rider imparts when on the bike, over the bars when turning, over the rear when under power, hanging off to initiate a turn etc., same with the boat. Especially a sib. The longer the sib the less rigid it is.......tubes may be larger but the material is still the same gauge. And so it is that it will need more input (weight transfer) once underway.
A 5m sib is bendier than a 3m one, but this can be overcome with crew weight transfer.
You are already noticing the reaction of the boat to the conditions. Your crews' reactions need to counter those very conditions.
If it means getting your passengers (crew) onto the bow to get you out of displacement and onto plane the get them to move forward and bring them back a bit to balance your vessel once planing. It's all good fun, involving and effective. Essentially it's an organic process using your feel for what's required by allowing your "mobile" ballast to trim the vessel. Transient ballast is a very expensive commodity in the most expensive of craft and you have it here for free ( unless your mother-in-law refuses to budge).
Main thing is to get everything that's standard, correct. Pressure ( and do it right, on the water), engine (weight,prop,shaft and trim)..........the rest is moveable. Find a medium, fix what you can, tanks etc., and let the crew know that they are part of the boat and not just passengers. You'll feel the difference and so will they.
And if you're singlehanded then get a tiller ext and be prepared to get yourself up to the bow when you're WOT. but be ready to scramble back when you've gone over the top.
Weight transfer. Pressure. Rigid vessel.