I note your decision not to tow on US 550 and expect it is a wise one. However, one can head south at Montrose for a visit to Ouray without any difficulty whatsoever. Of course, that means heading right back out north to Montrose to pick up US 50. The difficult portion of US 550 is on south of Ouray toward Silverton and Durango.
I have no idea whether this would fit into your plans at all, but it is just an additional bit of information.
Ouray is one of our favorite towns to visit in Colorado. Several good RV parks including a state park conveniently located.