Actually, your change makes a whole lot of sense to me and we have been RV'ing for over 52 years now and had every type of RV except for a toy hauler. Have a carriage Carrilite 5h wheel and an 11'4" Lance TC presently and had 3-4 different RV's at the same time for decades. We are 75 years old now and the 2 are enough to do all we want to very comfortable. The TC is for spring, summer, and fall and we use and the 5th wheel for towing to Florida for our time there in winter for a few months. My wife is disabled and can only walk short distances and minimal bending so slipping on snow or ice in winter at home is just not on the menu!
With a TC rig, you can take anything along if so wanted for some trips that you do with your toy hauler by just adding a simple inexpensive enclosed trailer which is so easily pulled behind your TC/DRW truck rig. Want to go on a day trip while camping? Just leave the enclosed trailer back on your CG site!
The TC rig give you so much more versatility and can park in normal parking lots when going to the stores etc. Also gives you a home on wheels with you if the weather gets bad while away from your CG site and you have all your clothes you'll need plus food and shelter and even a place to flop for a little nap wherever you so desire. All the same appliances as in your toy hauler and also an onboard generator for 110 AC with no shore power if you buy with one built in or an external generator as that works too. Better fuel mileage than towing the toy hauler and no issues getting to the fuel pumps and back out as you are about the same as your rig is about the same size as a pickup truck.
Our first RV was an 8' TC on a 3/4 ton truck purchased new in early 1965 and for the last over 52 years we have always had at least one TC owned at any time. Even though we had TT's or 5th wheels or class B or a class C or even one of our big Allegro class A MH's at the same time. None compares to what all can be done or with/in a TC rig. No problem launching a boat on a trailer if you take one along with a TC rig.
We will not be without a TC as it is so great for long driving trips like going to Alaska etc or cross country trips or 1 and 2 day trips and/or being on the road most of the time and where pulling a large trailer just to sleep in at night is crazy as you need a CG and usually some AC. Not needed with a TC rig as you can park them about anywhere and no one would even know you are inside sleeping. Small to heat or AC yet has all the conveniences of a larger RV.
I'd say, GO FOR IT!