Yes, size matters. It can be too big for your vehicle to tow, and still be too small to live in. There are compromises to be worked out.
My wife and I went through this in 2004-5. Mid/full size SUV that could tow 6000 pounds, looking for a RV for short campouts and road trips. 18 months of working out what we could tow vs. how much living space she wanted, we gave up on the trailer solution (needed a new larger tow vehicle) and bought a 30 foot motorhome, living space equivalent to a 24 foot travel trailer plus extra king or queen bed.
You have to figure this out for yourself, as the big question is "how much space do I need to live in" rather than "what can my truck, minivan or SUV tow." Go to RV shows. Get into RVs of all different types and sizes, act out living in them, what will everybody be doing at what time of day.
I'm in a RV club that's been active for at least 20 years. People who work out this living space issue will buy a RV that last them 10-20 years, until needs change from aging or family size changes. People who don't work it out have bought as many as three different RVs withing five years. Getting the right RV to live in is a whole lot more important than getting the RV your current vehicle can tow. If you find a towable RV that fits your needs, you may be looking at a different vehicle to tow it; or, it may be self propelled.