The surprising thing to me is how many of these roads mentioned above I have been on. Utah 12 was mentioned (Bryce Canyon to Torrey). Actually, it is one of my favorite drives. I will have to agree with the road over Douglas Pass in Northwestern Colorado. That one required constant driving.
One of the worst stretches I remember is just north of Birmingham, north of the intersection where the South loop meet I-20. (I think it was I-59). So bad I turned off to find a non-interstate--any road other than the Interstate we were on. Maybe it's better now.
We know well that section of I-20 between Vicksburg and Jackson--especially right in Jackson. I've always wondered how the highway through a state capitol could be so bad.
The city I do not like driving through: Austin, Texas. I've made I-20 through Atlanta to I-285 and then north, many times; have towed in L.A., Memphis, Housto, Nashville, for example; but I try to avoid Austin.