I agree with the OP. He paid for it and it was his for the duration. When he was shown to the site and it was designated as his for two days, there was a perimiter set for his occupation. He was entitiled to come and go all he wanted through the rental period. Now, he could have left something in the sight, which usually works, but who's to blame when, upon his return, it is missing and someone is in his sight. Though not obligated, he could have stopped by the office and let them know he was coming back, but again, not an obligation.
The real frustration has to be that when faced with someone else in your site, which you paid for, the on duty clerk take a ho-hum attitude. As someone else said earlier, good customer service, with at least what appeared to be a sincere appology, probably solved this quickly.
BTT,
Write that letter to the CG owner. The owner may not know what is going on and that the service was less than desirable.