You've got it figured right. "Max Towing" is a marketing figure that assumes an empty tow vehicle, and even in that condition there may not always be enough "cargo capacity" to carry the hitch weight of a trailer that heavy.
For what you want to carry in the XL, you probably need to be looking at TTs with a GVWR around 6000 pounds.
I picked up a 12-pass van with "Max Towing" numbers around 6000 pounds, with the idea that the largest travel trailer I would ever tow would be about 4500, since the van is sometimes going to be carrying people and their stuff. If I wanted to tow 6000 pounds of trailer, I would have looked for a van with 9000 "towing capacity."
I think you can find a travel trailer under 6000 pounds that will be suitable for five people, if you concentrate on models with lightweight construction and are willing to have some of spaces converting from living space to sleeping space. Usually these are in model lines carrying names like "ultralite" or "superlite" where the attention to weight control extends to all the furnishings and might involve some more expensive than usual construction technology.