Yes, depending on what you're testing for. If you want the batteries state of charge you need to disconnect the shore power cord. But then you should turn on a load like the furnace and let it run for about 2 minutes first to remove the surface charge from the shore power.
However, if you want to check if your charger is working, take a battery reading with the shore power disconnected. Then plug in shore power and take another reading. If the charger is working you should see the voltage jump up to at least 13.5v.
