As pointed out I doubt you will need to stay overly close to the swimmers. In my area we often fish for sword fish in 600m of water, at these depths we can't anchor so put the engines in and out of gear all day, day after day.
We also do this in shallower water targeting snapper and photographing wildlife close to washes. Even driving our boat onto the trailer involves plenty of in-out of gear to keep the speed down. We just make sure the rpm has dropped down (warm engine) and keep the gear change quite brisk rather than slowly clunking it in.