I'll be the first to say I use my Chevy Silverado 3500 Duramax Diesel Duly as a daily driver. Sure, it uses more fuel than our family car, and diesel fuel has cost more than gas for several years now. For me, the trade-off IS THAT comfortable tow.
You really need to think in terms of what is the ultimate extreme usage you'll be experiencing with your truck, not what is the lightest usage and hope it's OK when the extreme happens.
The other good thing about my truck is, I live in Central Indiana in the country, live 25 miles from my office and depend upon this truck for traveling back road over some adverse conditions too, like snow and ice in the winter when my family car is grounded in the garage! Also, I use the truck for hauling a utility trailer, and it serves a lot of purposes, and hauls a lot of "stuff".
Here's the thing.... if you're even remotely thinking of diesel / duly .. go for it. 1), you won't regret it. 2) your finances will adjust. 3) you'll find it so much more enjoyable towing (anything), and 4) they are just plain fun to drive!
If you talk yourself out of it, you'll probably be kicking yourself later, always wondering "Why didn't I get it when I could have?" You'll never have peace of mind wondering (would it have been better IF?)