How do you feel about people walking through your campsite?

We had a lakefront site and had left to go to the store. We returned and we relaxing outside when a couple came up from the water dragging their kayaks. My husband asked if they had a problem while out on the water. The said "no, our campsite is right there (pointing to their site across the road) and we don't have lake access from our site". Hubby said "there is a public launch area about a quarter mile from here with places to park." They said "yes, we know but this is much more convenient." Hubby said "next time reserve waterfront then but you can't access it from our site that we paid for. Don't come back again or I'll ask management to explain it to you". They left quickly and unhappily but didn't try it again. The sites were narrow and long with trees on the edges so no way for them to access the water without walking right by our camper, chairs, table etc.