I pulled a swivelwheel-type trailer for a few years behind my 35' 5th wheel. It wasn't bad at all, but I wouldn't want to do it behind anything any bigger. It adds quite a bit of rear swing to the trailer.
As for the removing too much weight from the pin...there was a poster on here who had severe handling issues with his unloaded toyhauler. When he pulled it loaded it was fine, but unloaded he had severe sway issues. It finally was returned to the builder to be modified so he could have more pin weight when unloaded.