Yes, parents take their kids, but even worse are the school groups. We went twice (yes, dragging my kids :-) ) The first time was the day before a hurricane. Once the town reopened, we had an amazing time. All the school groups left, leaving my kids among a handful of kids there. They got a lot of personalized attention.
My kids did not need to be dragged, BTW. The minute we left, my daughter started begging to go back. She was competing in local spelling bees, so I told her if she ever made it to The National Spelling Bee, we would add on a trip to Williamsburg. So, she did.
The second time was in May. The place was awash with bored school kids. It was hot and humid.
So, best go right after a major hurricane, if you can do it :-)