how does a battery boost switch operate?

If it is working right, it operates a big solenoid switch that ties the two battery banks together (engine battery and house batteries).

It is fairly common for the solenoids to go bad.....as they can carry a LOT of current when used.