While I do recommend you get your rig weighed, as you are likely on the edge, I'll share some of my own experiences when towing at or beyond my vehicles limits. I had my set-up nearly perfect and it towed great most of the time. However, there were times that the wind would be just right, and not super heard, perhaps the direction, that made it a little squirrley. I could go 1,500 miles and it would feel great and then one day for part of the day it would not feel good.
It is entirely likely you are on the edge and just occasionally it feels like it because of weather/wind conditions. Get it weighed and find out. Good luck.
I since have upgraded trailer and tow vehicle and have extra capacity in payload and towing. Always feels good now.
p.s. Anytime I've weighed my rig or weighed my tongue alone for the first time with that rig/trailer, I've always been VERY surprised. You might be as well.