I work in the auto industry, and one of the biggest problems I see with diesel trucks are people that buy them but never use them the way they should. People have a trailer/boat/whatever that they pull a few times a year, but use it as a commuter car the other 350 days a year. Short trips without a load and not giving the engine a chance to warm up will cause you many problems with the turbo, fuel, and exhaust system.
The other issue will be maintenance costs. Diesels are more expensive to operate and maintain.
They are great if you truly need a diesel pickup, however, based on your camper size and how often you pull it, I don't think you need a diesel.