Towing w/out weight dist -- am I being too conservative?

If I were you, I'd tow it to the other place (or to home) on the ball, although I'd keep the speed down to 50 or so until I knew it felt stable. The brake controller is right at the fingertips if it were to misbehave, but it probably won't because you have enough hitch weight.

The WD hitch will serve as long-term peace of mind, though. And it will reduce bouncing (improve your ride comfort). So I'd still install it, or get it installed.