I have never had an RV...I always thought having to "break camp" every time I needed to go somewhere, once we were at our destination would be a pain in the butt... BUT, I also imagine that if you towed a smaller vehicle around, you would be accomplishing the same thing as we do with our truck, but more economically, of course.
As in all things, it depends on your personal preferences. If you can afford an RV, you can probably afford a "toad". I have looked at mid-line rv's and was astounded at the price tag. My dream, when I retire in 10 years or so is to get me a nice/decked out 5th wheel and a nice F-350.
As it stands right now, my TT and PU fits me and my family just right. Do what feels right for you. IMO.