It very much depends on the fridge. There's no real need for it to be consuming power when off; on my (Dometic) fridge, I'm pretty sure that the power button does cut 12V power to the fridge controls. (Oddly enough, it does not cut power to the interior light in the fridge for some reason.)
There may be other things on your fridge DC circuit that are consuming power. Is your TV antenna amplifier turned off, for example?
If your battery is indeed being properly and fully charged and is running out in a couple hours with a 650 mA load, you have a bad battery. It's not unknown for new batteries to be defective, though thankfully it is the exception rather than the rule.