With our solar setup, we turn off the inverter if we are not using the solar system. In other words, it's off until we need to watch TV or use the outlets. While the inverter is off the system is charging and storing energy. We found that by leaving on the inverter, which we did for years, our batteries were not keeping up. Now that we've wised up, we have plenty of power when needed.
When we're home and plugged in, our inverter is still off and yes, it still charges. As you noted, when we are en route, we turn on the inverter and the fridge runs on solar while we're traveling. If it's cloudy and doesn't charge adequately, we turn off the fridge and rely on ice to keep it cold till we stop and turn on propane. We do not have a residential fridge.