I suggest late May because June doesn't necessarily beat the crowds. If you are from the north you think summer break = mid-June to after Labor Day. In much of the South it starts in May, and we start filling up those family vacation destinations before you all even get to final exams.
I've visited in March (a long time ago) and December (2003) and both were good times.
Bugs? Shouldn't be much of a problem in the historical park complex, no matter how muggy it gets. In the area campgrounds, from late March through late summer (which can include most of September) there will be plenty of bugs in swampy and wooded areas, and you'll be aware of them. Bugs and bug-carried diseases is why most of the plantations moved upriver to the Piedmont, and wealthy folks had summer homes in the mountains. Controlled today by poison in urban areas, still present in pockets of semi-wilderness, all the coastal lowlands, not just James River area.
Two years living in the area, working outdoors in the wet areas :(