I appreciate everyone's responses so far.  Putting ballast on the pin would not be my first choice, just wondering if it would work.  As well as wondering how much I am putting on the axles being a little low in the back.
My pin is really light.  
Pin garage empty: 2,620 lbs (17.75%)
Pin Bike+water: 2,140 lbs (13.16%)
With the bike and water it tows absolutely wonderful.  Without the weight, it sways a lot.  
I was restricted by length due to my choice of camper storage.  As a result, I chose a double axle trailer under 38'.  I knew there would be limitations but this model seemed to have the best CCC of the ones I looked at.  Once on the scales, obviously I am having some issues with distribution.  It's fine for my cycle and smaller sxs, which is what I will have to stay focused on.  I will play around a little more, run less fuel in the cell.  Move some weight around the best I can, maybe see if lowering the pin will help any.  The factory spare is just in front of the axles, maybe I can move that to the front storage as well.
Upgrading the axles seems like the best option, but not necessary if I can figure out how to distribute the weight.
I'll keep everyone updated.