There is plenty to do in Richmond, it was the capital of the Confederacy after all. Must see are the Confederate White House, Confederacy Museum, Tredigar Iron Works National Park, and Monument Avenue. A short drive away is Cold Harbor National Park where Grant's frontal charge resulted in thousands of casualties in only 1 hour.
Also in town is St. John's Church where Patrick Henry's "Give me liberty or give me death" is reenacted.
Little known is the fact that Edgar Allen Poe lived much of his life in Richmond, and these is a small but sensational museum in Richmond (Poe Museum).
I recommend stopping in Richmond as there is much to see.