Let it go. Don't take the attitude that you own the site. If the large group is walking around the perimeter of your site don't let it bother you. Focus on something else.
Unfortunately you were surrounded by a large group of friends and you were the odd man out. Why I don't condone their behavior I would not let it ruin my stay.
Eventually you became obsessed with the idea that they were trespassing on your site which may have been correct to some degree but nevertheless. Is it really a tragedy if they were cutting through?
The only reason the cutting thru ruined your weekend is because you let it.
Again if the weather was good and things were decent otherwise.
Just ignore the rude behavior vs.letting it ruin your day.