Getting away from the LJ vs BA discussion for a mo (the self righting properties of a good LJ should never be overlooked though).
I am similar build to you and find that the fit of a BA depends entirely on what other gear I am wearing at the time, with just a t-shirt then a medium is good but with a jumper and waterproofs on I need an XL.
You could look at one of these:
Which are genuinely one-size-fits-all in the same way as an inflatable lifejacket, ie if you adjust it properly.
NB Most of the lifejeckets/buoyancy aid models we stock have been tested in the water by me, partly because I do not naturally float so they are a very welcome bit of boaty kit!