I moved our PUP around all the time with our riding lawn mower. I put a ball in the hole and it worked perfectly fine. Did this for 6 years. I now have a John Deere (not a gigantic one, just a simple riding mower), and I still keep a ball in that hole to move my utility trailer around the yard, or for hauling smaller loads from one side of the property to the other.
About the position of those pavers, just move them where your camper sits now. In a few weeks the grass will grow back and you'll never knew where the camper originally sat.