Your own idea of aluminum plate is exactly what I would do.
I also worked in the marine industry for a while 25 years ago and when we attached an engine to the transom of the boat we would put silicon around the bolts if we ever had to take the engine back off you could lift the boat off the trailer before the silicone would let go.
Most people aren't aware of the holding ability of silicone and its merit as a glue.
So as said I would use a four by four piece of aluminum drill holes for ladder then counter sink them from rear use machine bolt from back with a nut on ladder side, cover back side of plate with silicone use a #10 screw in each corner when the silicone set's there will be no coming off....