Trac Bar on the rear should have taken away the side push on the rear.
It sounds like the "Tail is wagging the Dog" when the Toad is attached.
Several others have observed / suggested to have your RV weighed to get the tire pressures correct for the actual travel weight.
I'm thinking (as others have stated) that you don't have enough weight on the front axle in relation to the rear, This allows the Toad to steer the RV.
Another thought what is your ride height UN-loaded, fully loaded w/ & W/O the Toad ? (measure the 4 corners top of the wheel wells) Are you squatting Down in the rear a excessive amount when fully loaded ?