This only happens to me at a certain state park I go to. The site I reserve is very large and the one next to me is just a sliver of land next to the road. The last time there a group came and unloaded their gear and started to setup (tents). The next time I came out they were setup like 10ft from my fire pit so I told them they were about 20ft on my side. I told them I know their site is small and I dont mind them coming over on mine a bit but they need to move over.
I told the the sites usually go from power pedestal to power pedestal. They didnt know and moved their stuff. They were over on my side somewhat but it didnt matter to me and we all had a good time.