I'm biased towards 5th wheels mainly because they tow very well. I've pulled through some pretty strong crosswinds and never felt uneasy. They are taller than a trailer, but I also like that because they feel more spacious inside.
As far as it being too tall for your pole barn, are you buying the toyhauler to store in your pole barn or to use on the road? I would buy a toyhauler based on how it's going to be to stay in, not how well it stores, unless you only plan on using it a few weekends, and maybe a couple week trips. Then I would rank storage ability pretty high.