Wife and I didn't start camping without the kids until the last couple years, when the kids were old enough to be home alone. We didn't have relatives close enough to easily watch the kids for a weekend.
We have greatly enjoyed camping as a family. Patience is always required, plus the ability to find the good in everything, plus the willingness to change plans at the last minute. Having said that...
I absolutely LOVE camping with just my wife. We usually go for more rustic campgrounds without hookups, and plan fancier dinners - cleanup is certainly easier with just two people. We go for hikes (slower now with my aging knees) and/or short touristy trips to check out the area. It is fun and relaxing, and nice to know the wife is still willing to put up with just me for an entire weekend!