The way that my batteries are wired in my lance for the 70 watt solar panel is that the hot wire from the panel goes to the battery side of the cut off switch. This is so that the solar always in hooked to the battery even when you shut off the power to the camper. This allows just what you found, you can shut off all the draws for power from the camper and still charge the batteries from the solar.
I did my wiring based on info from my dealer RVSolar, in Arizona.
A good controller will keep your batteries charged as long as the sun shines.