Our 11' TC puts my dually 1,600 pounds over the GVWR with the family loaded up and ready to camp. We're a family of 4, the two kids are 6 and 3 years old. We don't have enough room and I'll likely be selling the camper to get something larger.
4 adults in a camper that you can put on a SRW truck isn't my idea of a good idea.
I take my dually off road frequently, I've been doing it for 11 years. Not hard core rock crawling but rocks, ruts, mud, slopes, etc. It does just fine. I got a rock stuck between the tires exactly once. That was in my driveway.