We have done the trip down from D.C. to Williamsburg many times with no problems. Don't travel during rush hour to and from D.C. to Richmond and you will be fine.
Mount Vernon is just south of D.C. and you really can't avoid 95 there unless you take US 1 which can be just as crowded during rush hour and full of traffic lights. Timing is everything in this area. Weekends are better than weekdays because there is a lot less commuters on the roads to D.C. and Richmond.
To go back to MI, if you want to avoid D.C. completely, take I-64 across to 81 North and take that up to any cross route you like. If you are going by I64 stop at Charlottesville and see Thomas Jefferson's home - Monticello.
