Have used 59 litre Ortleib dry bags for holding water as ballast. They dont do any damage to the boat, and if they should come away when punching through a wave, arent likely to do break your crew. (they 'sploge' around things, rather than smashing them).
When you are running with the sea, it's easy enough to trim the boat by empting the bags and folding them up. On days when you dont need the ballast, ideal for keeping stuff dry.
Only down side to having several black bags strapped to your deck is that customs seem to get interested.