I think the first step is to determine what the internal wiring of the switch is. My guess is that it's a single pole double throw switch (presumably momentary), so there's a common connection, a normally open connection, and a normally closed connection.
If that's the case, I would guess that the common should be wired to the solenoid control wire. The normally open is connected to +12V, probably from the house battery. The normally closed one is probably connected to a switched lead in the chassis somewhere so that the solenoid closes while the engine is running and permits the house batteries to be charged by the engine alternator. It may actually be a little more sophisticated circuitry than just a switched-by-the-ignition wire, but the general idea is similar.
If this is indeed the intended setup, then when you have the engine running, or perhaps just the ignition in the on position, the relay should normally be operating to connect the two batteries together. If the ignition is off, the relay should only operate when the aux start button is pressed. Staying closed in this case implies that the switched chassis lead is not wired properly or whatever is controlling it is not working properly.