We've driven it many times with our 40' motorhome towing the Jeep. But then, we enjoy these kinds of drives.
It sounds like you're an experienced driver with big rigs so I doubt you'd have a technical problem driving it. However, if heights bother you - or your passenger - the alternate route is going over to Cortez and heading north toward Telluride. It's still a mountain highway but a tad milder!
We also love staying in Silverton and doing the awesome 4-wheeling routes out of there. Durango to Silverton is no issue. The part some folks don't like is from Silverton to Ouray. To us, Silverton is a more scenic and fun town than Ouray.
Have fun! It's one of our absolute favorite areas.