It doesn't bother me if folks in a spot next to us pull in late ... if they're in a small motorhome, powered by gas instead of diesel, and don't have to level or make any hookups before turning off their lights and going to bed.
That's how we would arrive late in a CG so as to not disturb other campers. We try to arrive with full enough batteries, a full enough fresh water water, and empty enough black and grey tanks.
Jockying around to precisely position a towable, and/or leveling, and/or hooking up ... can be done in the morning after other campers are up.