As noted, the magnet is supposed to wear/rub on the hub, that's what activates the whole shebang.
According to what I've read in several places, when the pads are down around 1/16" or less they should be replaced. The figure most often quoted for new thickness is 3/16", so if your shoes are at 1/4" you've got quite a ways to go before they're worn too much. I certainly wouldn't be replacing my brakes with 1/4" brake shoe material left.
How many miles have to got on this trailer? I have a little over 20,000 miles on my 2010 triple axle and the brake shoes look fine.