You definitely have too little tongue weight. Your description of it being fine when empty, but when you load it up (toys in the rear) it gets squirrely tells me it is the tongue weight. As a former manufacturer of equipment trailers, my analysis (without actually checking it out in person) is that the axles are too far forward for the trailer when fully loaded.
A little extra tongue weight won't hurt anything as long as the TV can handle it. Too little tongue weight can get you, or someone else, killed.