My experience has been very similar to yours but my son was a good deal older. We got out of RVing for multiple reasons-our son was in college and the extra money came in handy, we weren't RVing due to crazy schedules and we got a little tired of repeating the same vacations over and over due to time constraints. Living in a polarized area of the country makes getting lots of places hard.
Don't let anyone tell you that if you don't RV you'll miss prime time with your son. That's pure BS. It's not the what but the who and time. Weekends may be better spent right now doing day trips to the zoo, amusement park, beach, etc. or even just playing ball in the back yard. Much less planning, less stress and just as memorable. And there's always renting a cabin or an RV for the weekend. My son loved to RV but wouldn't give up the trips to Disney World, Yellowstone etc. for anything (RVing and plane travel are NOT mutually exclusive). Finances aren't even relevant in this discussion-if you want something bad enough you will find a way. For you it appears it's not an issue. One thing I did discover though we did feel a bit of "guilt" so-to-speak when we didn't use the Endura but took a vacation. Selling it took a good amount of pressure in a number of ways off our shoulders.
Size doesn't matter if a unit is just going to sit and not be used. Like anything mechanical inactivity causes a whole new set of issues over the long-term. Ultimately you have to decide for yourself what is going to stress you the least amount.