I don't think they are "just used to the sway" but I also don't completely buy it, however there is a lot of truth there also. Some trailers are naturally more stable than others and that impacts each person's experience as well. I know a good sway control and being properly set-up does help a LOT. I used to tow a 26' 5600lb trailer with my Buick wagon. It was great on a test tow but once loaded and on the express way, I was getting blown all over. I tightened up the friction sway control and it was then controllable and felt safe but never wanted to just go straight down the road. The friction sway control was always sticking in a non-straight ahead position, felt fine on the curves or non-straight roads, but very straight roads it would wander left or right. I then upgraded to a Reese Dual Cam system and that was a huge step-up. Tracked nice and straight and even better control over passing vehicles and wind. Later I upgraded the tires from 245/60R17 all seasons to high performance SUV tires in size 235/65VR17. That also was an improvement. It hauled the trailer just as nice as my current 2500 Suburban and larger trailer except on the windiest of days (then the Suburban is better, even with the larger trailer). The wagon should have been a poor tow vehicle with only a 116" wheelbase and a HUGE rear overhang. But it did pretty good, we towed about 8,000 miles with that combination. Rear axle was way over loaded as well (lots of extra tire capacity however).
Based on my experience, a good sway control did make a huge difference but the tow vehicle makes a huge difference also. I only towed my 26' trailer once with my Suburban before getting a larger trailer but I didn't even bother to align the cams on the sway control or set it up properly and I literally couldn't feel the trailer back there and it was a windy day.
I've also towed a 5th wheel in the past and that was definitely a great experience. Most days my travel trailer feels as good, but not every day. I still feel the wind and trucks a little and I don't ever recall those feelings with the 5th wheel. I'm very happy with how my trailer tows and I wouldn't get a 5th wheel just for the better towing experience because I think with the right tow vehicle and good sway control the difference is small.