2 days would be my max at Crater Lake. If at all possible take the lake boat ride. You like to hike, well it's a pretty good hike down to the boat dock. Way more to see down on the water while cruising around inside the crater.
The Rim drive is a must also.
There's also a really nice hike to the top of Mt Scott. Fantastic 360* view up there.
If you don't mind dry camping for a few days I would try getting a spot at Diamond Lake CG mid week. If not then Broken Arrow has plenty of sites. It's only 15-20 mins to Crater lake entrance from there.