I occasionally get hit with the same effect. The odd thing is it is not constant.
Some days it is a noticable push, some days it is not there. Since I always set the cruise at 60 mph and hitch up the same way, I just attributed it to a shift in the TT loading due to changes in the gray, black and fresh water tank levels or perhaps a cross wind condition.
Though it can annoying at times, it is one of the variables encountered when pulling a TT. I have a half ton and a 26 ft trailer, the effect might be worse with a larger TT.