You don't necessarily get a 4WD truck because you want to go off road.
You get it because you can't get going uphill on gravel with your trailer on, or because you live or travel where it snows. If you live where it ever snows and you buy a 2WD diesel pickup, congratulations on owning the worst possible vehicle to drive! Tons of weight up front for the rear to push, and very little on the rear to get traction. It's the worst possible situation.
Also, many mountain passes require you to chain up if there's snow, UNLESS you have 4WD.
4WD works great for driving up onto leveling blocks without spitting them out from under the tires. It helps you cross grass or other surfaces that you're trying not to put tire ruts in.
Plus resale value as mentioned. In my mind - and I'm not alone - a truck without 4WD is a deal breaker I wouldn't buy at half the price.
The payload decrease, seriously in the real world, is imaginary. The extra weight is all up front where you could use a little more anyway when you have a trailer. There's no trailer you can't handle because of the slightly heavier front drive components.
If you live in the southern US where you never get snow and you never drive anywhere but into and out of paved campgrounds, that's different. The thing I always notice when I drive through Southern California is all the 2WD pickups!!