Mackyb, if you unhook the ATV trailer does the TT tow OK? I suspect that your TT tongue weight is border line trying to stay within your weight limits. If 2 stout guys can move the TT's hitch around, your to light in the nose.
First order of business is to get the weight balanced. You can search on how to use a bathroom scale to get your trailers tongue weight.
You don't need a diesel to pull your set up. A longer wheelbase will help. A big block with 3.73 or a small block with 4.10 gears will handle it fine.
For 10 trips a yr, well what happens if you only get 3 1/2 trips in. You need to be safely equipped to make it back home in one piece. Don't go falling off the mountain roads.