Annoying, but I guess I have more important stuff to worry about...I could see it as situational too. As mentioned in a couple of posts about access to river or trails. I've come into a campground on a trail and found myself in a campsite before. Poor design, but just went around the edge of the site and on my way.
Now more annoying is the response I get when I try to teach my kids not to cut through other sites if it's not a site where they're already playing or friends of ours. Kids are pretty responsive, but other adults obviously don't think it's a big deal.