The tongue does not have enough weight.
You said it was acceptable empty.
Then you load it. Just like the see-saw in the playground, the more weight you put on one side of your axle = less weight on the other side. You need to adjust your load so the heavy side it toward the front of the trailer. Once you get the balance corrected it will follow you happily.
You can test this. Add weight to the forward floor area. Put a piece of carpet down and put some cement blocks in or my personal favorite, use cases of beer. 20 cases will give you 360 lbs. You might need another 10 if it's going to be a long weekend.
I once pulled a utility trailer 300 miles. It started wagging at 30 mph, I stopped 3 times moving heavy item forward each time. Each time the speed that the wagging started at went up, until finally I could travel the interstate at speed without any problem.