Why is towing my little trailer so stressful?

I agree with Doug 33, a better hitch setup would not hurt. Another thing you want to look at is weight. You say it's only 2500 lbs. but by today's standard that's a heavy camper for it size, plus the extras like food drink gas personal belongings; adds to it.
Whats the TV gross combination weight rating and what's the weight of the TV itself. Better make sure your not exceeding your gross weight ratings. that will make it wiggle too. normally you can find the weight specs on the door post on the drivers side.