That's exactly the way such things should be handled...the newest folks should gracefully give way to those who were there first.
As for me: I love to camp near families with children, even when my own Grandkids aren't with me. Watching kids have a good time camping is great entertainment as far as I'm concerned....and the more kids, the merrier!
I do have a sort of weasely "Grandma trick", though. I always keep a bunch of Dollar-Store glow sticks in the trailer and pass them out to the "neighboring" kids as a sort of good-behavior bribe. They're always tickled pink and I never have any problems with misbehaviors at/near my site.