The condition that you describe sounds like "tail wagging the dog". The Toad is pushed and the rear of the RV sideway on the axle which causes the front end to point out of track, thus the correction. Normally the TracBar will solve this issue and locks the axle to the chassis. You stated you have a TracBar so this shouldn't be an issue. Perhaps check the connections on the TracBar to make sure they haven't come loose? Your sway bars have no bearing on the issue.
Initially I had a similar issue (without the Toad) and it was mostly caused by alignment of the font end. Perhaps a place to start?