The only downsides I've experienced is related to boondocking.
Unless specifically built for it a trailer doesn't want to go off road. For a short distance or on softer ground you might get away with it, but no standard TT or 5r is going to survive getting dragged two dozen miles down a rocky Jeep trail. I love wild camping and feel my trailer holds me back.
Second is stealth or urban camping. A trailer really sticks out, even compared to a motorhome. More and more communities are banning RVs parking on streets because they're afraid of poor people. Tourists trying to visit get punished as a result. A self contained rig can better fit in private parking lots, and is more mobile in general.
2013 KZ Sportsmen Classic 200, 20 ft TT
2020 RAM 1500, 5.7 4x4, 8 speed