You are going to have to leave the battery switch on. Quite a few of the items in the unit operate off of 12 volt supplied by the battery. Your refrigerator is one of them.Any lights you don't plug into an outlet are also 12 volt. When plugged in the converter charges the battery and may supply a small amount to power other stuff. Any time you use a high draw on the 12 volt side, like running slides in or using your jacks to raise or lower the trailer, you are using the stored energy in the battery.
If you are not hooked up to electric and you are not using the unit for several days, that is when you would want to turn the battery switch to off. It would prevent parasitic loads from draining your battery. I say this assuming you would not have anything in the refrigerator and would not need it running on gas for that time.