it is a valid concern if you are a 'backwoods' camper as you described yourself. Think of your current camping, did ya pull in with your current rv and say....wow bet I could not have gotten into here with something 30 plus feet? :) Your camping and past places can give you an idea of if it will be easy or not for you to increase your size. The height of the 5ths can be rough also.
we went from a 40 ft 5th toy to a 34 ft (total length) TT toy. We love being smaller. But the way we rv now, we know our state parks and we know we can easily fit our 34 ft TT toy....when we retire we will probably go a bit smaller so we can do more 'backwoods' type camping. We want to be able to fit everywhere :) best of luck to you