As you back and turn, your rigs wheels will be in the tightest corner, and then your back wheel and your front wheels. So when you set up to turn into your site, you want to have your rigs wheels closest to the inside of the turn.
So, to answer the question, pass the site and have your rig on the side of the road closest to the campsite. As you back up the truck will need to go to the side of the road furthest from the campsite.
It might help to imagine driving into the spot truck first. To make the curve, the truck has to take the outside curve, so the trailer wheels don't run over the inside curb.