Unfortunately for what you're facing, the caster is going to work against you. While caster is a name for the wheel, it's also a behavior built into the wheel. If you look at a shopping cart you will notice that the wheels behave the same way your tongue jack behaves. That's because of the angle of caster built in. The front end of your car also has caster built in and it's why the steering wheel centers itself as you come out of a turn and keeps the car going in a straight line when you let go of the wheel.
If you wanted your tongue jack to not pivot as you move the trailer you would have to find a wheel where the axle is directly beneath the jack tube.
It might be easier to get the trailer lined up and try to push it where it needs to go. If the caster tries to spin you will need to kick it around 180 degrees so it will track straight. If that's not possible because you need the wheel physically in a different spot, then another option is to load the back of the trailer so the front becomes lighter. This won't stop the caster swing, but at least after you get the trailer where you want it you should more easily be able to spin the caster wheel around where you want it to sit without the trailer moving.