I don't have a toy hauler, but for many of the same reasons you've pointed out, we've considered them for years now. When it finally came to our last decision (about 3 years ago), the conventional trailer won again. But, we also don't have the kids any more, and we don't have all the pets any more either. We had a bunch of pets... I mean a bunch. We're down to one dog and one cat, and occasionally one grandchild.
The attraction for the toy hauler was all that open space in the back. And the double bunk style beds that raised and lowered was something we envied. We don't have toys, but the idea of lowering the tail gate, setting up a porch, or putting a screen across the open back with an entire wall for a window was VERY attractive.
I (not my wife), really liked the bunk space over the garage with the opening in the living room. But in all reality, without kids, what good would that space do. It might be OK for storing stuff, but we've down sized so much, we don't have all the "junk" like we use to.
When it came down to the final choice, convention trailer won. But it had surround windows on the rear, giving us the view we always wanted.
If you are wanting the bunk beds for the kids, you have to ask yourself (and maybe even the kids too), how long do you expect the kids to be camping with you. I don't know your kids ages, but when they turn 15-16 years old, they'll begin driving and getting job. Their attitudes change (and not for the better), and the last thing they want is to hang around parents. The camping days are pretty much over for the kids when they turn 16. And then they get VERY interested again when YOU become grandpa and grandma! It's amazing how interested they get again!
Without knowing your kids age, or if you have plans for when the kids no longer travel with you, there's nothing wrong with getting a toy hauler anyway, even if you have no plans for hauling toys.
While you do have kids, it would be great for bicycles and such though. Once the kids are on their own, you have that space in the back to do with, whatever you want. Turn it into a hobby room, or a "man cave" or a "craft room".
One thing you do need to consider though, which is another reason why we finally opted to go with a traditional trailer, is because you need space behind the trailer to open the big garage door. On a beach, sand dunes, the desert, or the middle of a corn field, that would be no problem. But far too many campgrounds don't provide the space behind the camper to open that big door, because of trees, or simply, you're backed up to another campsite. This is something to think about. We realized most of our camping is in State Parks, and pretty much, there's few places we've been that had that extra space to open a big door from the rear, because something was in the way.
But, you have a great idea. Nothing wrong with it. In the end, only you can decide what's best for you. And I know, you'll enjoy it, if you go that direction! Good luck!