I'm not a morning person either. So I do my camp cleanup the day before departure. Then I can get up at 10 am, have a breakfast and be off by 11 am (or whatever the check out time). If I want to take a leisurely stroll around the campground, then that's one day I "need" to get up early while on vacation.
Also, remember that if you don't leave until 4 pm and someone is coming in, they may be setting up camp in the dark depending on time of year.
That's why most places make it between 11 am and 2 pm - it is the most convenient for both those coming and going.
Any earlier and it is a major inconvenience and even if you aren't leaving, you have to listen to those around you packing up at 6 am while you're trying to sleep in.
Any later and people are setting up in the dark and even if you aren't leaving, you have to listen to those around you packing up or setting up while you're trying to enjoy the afternoon/evening.
Having to break camp by 11 am - 1 pm is a minor problem when you think of all the other inconveniences to folks by changing the check out time. And at that time, you are least likely to disturb other campers who are late risers or early to bedders or who have small children taking naps, etc.