I think you'd be just fine. It's the same question we had, because we got to the starting point late in the day and were more concerned about "the unknown and discomfort" than anything else. We had the turn-around ace up our sleeve right up until we got to the latter switchbacks.
Here's the
story (you might get a kick out of it).
Here are photos from just off Hwy 550 up to the pass. A bit narrow in places. Maybe a three point turn on that one switch back spoken of before. I don't recall any clearance concerns for the segment you're talking. Overall there's plenty of width.






This is just below that switchback.

You can see far enough to pull over in a wide spot if oncoming traffic.

This is just an adrenaline moment.



Plenty of room to turn around at the top. Beautiful scenery.

Not very far beyond the pass is this view, where you start to descend onto the Telluride side, WELL before the one way road and the VERY tight switchbacks above Telluride where the road's real reputation takes over.

And again, much later after the above view you come to this sign.

Read the story. It will provide a little more insight on this segment you want to do - if I remember correctly. Just think smart, and I'd suggest a bit of tire pressure air-down. Just make sure not to air-down to where your later pavement run suffers. But high pressure (mileage improving) tires will be like marbles on marbles with that loose rock road. Much better to let the rubber compress and form around the road rocks a little. Just don't want low tires back on the pavement so there's a balance to consider.