Someone makes a rot treatment; it's similar to a really thin epoxy mix designed to penetrate the damaged wood, then cure up stronger than the original (unrotted) material.
I've seen repairs to glass boats where the transom cap was cut off (must be an interesting feeling to go after your boat with a chainsaw), rotted wood picked out, a repair solution was basically poured in to fill all the voids, and the transom cap rebuilt.
For yours, I'd assume, if you went this route, that you'd have to come up with some sort of form to contain the goo.
Or you could, as you're doing, replace the entire transom piece. Make sure you seal the entire piece, and seal any holes that are drilled later.
Have you thought about replacing the wood with a fiberglass transom?