I grew up in Visalia, and we regularly camped up in Mineral King (obviously tent camping). I think OP will be fine going to Potwisha, but no further beyond that with the 5th wheel. It would be different in a big diesel pusher motorhome, which doesn't bend, but a truck and 5th wheel should be alright. It's just after Potwisha that the road gets super curvy/narrow, and I wouldn't attempt it even with a small trailer.
I'm also wondering if people think Potwisha is furthur up than it is? It's only at 2100' elevation so it doesn't get the weather that the higher up campgrounds (like Lodgepole @ 6700') get.
Also, OP, even though it sounds like the decision is made for you, 41 is the easier road. My stepdad is a school bus driver for Fresno Unified, and he takes trips all the time up 41 in his school bus, and as a kid I went to winter camp at Sugar Pine and the charter buses made it there just fine.
If you wanted a more scenic drive instead of hopping on 99 to come down to Visalia, you could cut across Highway 180 and come down to Visalia through Reedley, or go even further east to Highway 63 and work your way to 198 that way. It will probably take a big longer than staying on the freeways, but it's much more picturesque for the most part.
2020 Keystone Montana High Country 294RL
2017 Ford F350 DRW King Ranch
2021 Ford F350 SRW Lariat Tremor