The visitor center and parking lot are down by the highway. The actual castle is on top of a high hill to the east looking out over the ocean.
They run buses up and down the hill all day for the various tours that are available. You could start early in the morning and perhaps do two or three tours by the end of the day if you had them all scheduled.
There is a state park campground about 5 miles south and then farther south down in Morro Bay are a bunch of private camp grounds.
Edit: Apparently they now allow you to schedule multiple tours up at the castle without returning to the visitor center. But they recommend spacing them an hour apart.
And there is plenty of RV parking at the visitor center.