we are currently downgrading. we are just like you....we do state/federal campgrounds over commercial and we need that smaller size for ease of getting a site and thru the narrow roads loaded with trees in every possible worst position :)
we have 40 ft 5th toyhauler. going to a 29 ft. travel trailer toyhauler. (34 total with tongue). losing like 5 ft. and some height to make it easier on us.
key is like everyone said, get the floorplan that suits what you want. size isn't as much a matter as long as your happy with the feel/comfort and more in it. We got everything we needed in our 29 ft TT toyhauler that we have in our 40 ft 5th toyhauler. just more compact.
best of luck to you in finding what you want!!