You won't need a class 4 or 5 truck to pull the toy hauler. And, you won't have a "monster" truck to have to deal with. A 3500 dual or single rear wheel diesel truck will be fine and will actually be a benefit and not a burden when you get to your location.
A toy hauler provides a lot of flexibility for what you will be doing. The garage can be used for an office or to carry a RZR or golf cart or other onsite transportation or both. I think it makes sense, but you need to decide what will work best for your specific situation.