Even if you don't intend to drive in mud or snow you may find yourself there. Or, simply pulling off the road into the grass after a rain you can sink in. If you are just doing state highways and campgrounds with paved pads you would probably be OK.
I don't go way off road but a 90 degree turn on a gravel road in the Amish area put a tire in the ditch. I would have had to disconnect and reorient (or get towed) if I didn't have 4WD to back out. Many a mountain road has turned suddenly seriously snowy, even in June. Quite a few campgrounds without hard pads have turned very soggy after an overnight downpour (especially problematic if you have to jostle back and forth in or out of a tight spot). Finally it is nice to leave the camper behind and go exploring on some of the back roads on Public Lands and other places.
It is a big expense and you can go without if you are very careful about where you go. If you can afford it, it is nice insurance and frees you to explore a little more than you could otherwise. An entire other can of worms is when you are disconnected and driving around with a vehicle with little weight in the rear end.