Depends what type of power is powering your fridge. Sounds like it runs on DC power but your 1-2 days gets warmer etc etc doesn’t really make sense.
In general, leave the battery on all the time. Save for when it’s in storage, generally, if it’s long term storage or you’re unable to make sure it’s not providing power to something while your away.
More importantly, learn the basics of what powers your fridge (and when) and what a converter/charger does and whether you have an inverter.
Similar question and answers going in another thread currently as well.
my first suggestion is use the internet to familiarize or train yourself in the basic function of RV electrical systems. Betting there is more than sufficient info, tutorials, YT videos etc.