According to Yamaha, the F150 max rpm is 5000-6000 which is a bit vague.
If you went down to a 17" you should make 5400-5500 under your light load conditions.
But with 8+1 on board you should find the motor will still be really struggling, a 15" will be much better.
Also, not true re propping for max rpm to get best economy. I had 19" on my boat which gave correct max rpm. I changed to 17" and although I could over rev the motors at WOT under most conditions, at a given cruising speed I was burning less fuel even though the revs were higher than with the 19". Less load on the motors.
Your boat is well under powered with 150hp, when the yammy goes bang stick at least 225 on there