Really, its all relative according to expectations and needs. Some want to tow a up a 6% at 70mph without running the RPMs high. Some are perfectly happy towing up a 6% at 45 mpg and 4000 RPMs. Some want to pass a line of vehicles doing 80mph (and don't deny it, I see plenty of 3/4 and 1 tons with 5vers doing this all the time). Others are happy letting everyone pass them. Some people need to tow in mountains. Others will never tow more than 200 miles from home.
People who claim you "won't have any fun towing" will often cite personal experience. That's fine except for the fact that more often than not its comparing apples to oranges. Lots of people touted how great it was to tow a 4000# trailer with a Chevy Astro. We towed a 2500# pup with one and it sucked rocks. Could be a lot of reasons for that but I never questioned those that had good experiences.
So would it "be nice" to have a big honking 2500 PU to tow a 4000# trailer? Sure it would. I have no doubt in my mind that it would be a fine towing experience. Is it practical to have the additional expense of one? Not for a lot of people.
The inline 6 in my Trailblazer towed our old 3500# HTT through mountains just fine. 50 mph or so at 3500 RPM. We tool along the flat roads at 62mpg or so. Now we no longer take really long trips. Our Trailblazer tows our new Jayco X20E, 4750# GVWR, just fine through WI. It really doesn't feel much different towing it than our old trailer. I wouldn't tow this combo through mountains, but I also don't feel I need to replace the truck for the kind of towing we're doing today. And trust me, I was concerned about towing this trailer with this truck, but it went well beyond my expectations. BTW, I have NO PAYMENTS. More money to spend on fun things. That said when we do replace it, it will be a bigger TV. We just don't have a burning need to do so right now.