The OP's point is why we try to look for out-of-the-way places to camp where families and other types of crowds most likely wouldn't be interested in camping. We even looked for these type of places when we went camping with our kids!
We look for boondock camping places, drycamp campgrounds, and campgrounds in unusual places - such as on Indian reservations. Nice, pleasant campgrounds with their various well-maintained facilities are always going to be a draw for the crowds. Look off the beaten track for the quiestest, laid-back camping spots.