Been to Rafter J, it's a wonderful campground, with a variety of settings to camp. Planning to go back there this year for a family reunion if we can get everyone out there at the same time.
Best times, like others have said, would be in spring before school lets out, or fall when school is back in session! I've been there in the spring and it was very comfortable weatherwise, and I have no doubt that fall would be equally comfortable. Since it's in the Hill City area, the climate will be a bit more comfortable than some other locations further east or south. The Black Hills area is usually 5-10 degrees cooler than in Rapid City, But it can still get warm there. We hiked/climbed Harney Peak when the temps were in the upper 80s in Hill City, and it was still a bit warm at ~7200 ft.