How do you feel about people walking through your campsite?

I had a fellow cut through our site many times one weekend. On one trip through I handed him my trash and said "here since you seem to know the way". He just replied "sure" and cut back through on the way back. Didn't register with him at all.