Whether the camper, visitor or CG likes it or not, the CG is liable for the visitors once the visitors enter CG property.
Fees are in place to discourage/limit visitors. They are also there to cover any additional expenses including insurance, created by the visitors.
It's not a deal braking issue, if you invite guest to your site, be prepared to pay the fee.
It's just not a big enough issue to be a deal breaker when choosing a CG simply because visitors are a rare occurrence for me. For the few times I have had a visitor I just paid the fee if required and moved on.
I can see why one may feel the visitor fees are unfair, but once you understand the CG is a business and the visitor always has an impact on the CG and increases their exposure the fees are easier to understand and accept.