You have a pretty solid base to start with, but I'd say you're taking a truck that is worth between $15-20k and putting over $10k into it, IF you do the work yourself minus the paint job, with the high likelihood of putting several more thousand into it (160k = high likelihood for injectors, front end work, etc) to have a truck that is worth between $15-20k.
No one wants to pay high dollar for a truck with a Maaco paint job and repop dash and seat covers. Shows it was not stored well to begin with.
IMO, drive it and take on repairs as they come or sell it and move up if you insist on "restoring" it. Strictly by the numbers. Leave emotion out of it, not a classic vehicle, and diesel trucks are dime a dozen. Well maybe 5 dimes.
Consider, after TTL, you can buy a brand new Tradesman diesel for high $40s.
$15k low end sale for yours and $10k you'd put into it = brand new truck for $20k or less.
Besides, towing with a stock 5.9 common rail mated to a 4 speed trans is light years behind any diesel in the last 10 years. For minimal out of pocket and better performance, I'd sell the 04 and look for a low mile LB7 with new injectors, LLY or LBZ Duramax before sinking money into the Dodge.
There are other reasons, given the choice that I wouldn't invest money in an early 3rd gen as well. Little stuff or personal preference.