Will a dirty magnet cause intermittent brakes?

Grease on the shoes is going to give you poor braking all the time it would seem. A bad connection could give you and intermittent problem. The brake connections under my rig used wire nuts. It's quick and easy for the people who assemble it but it wont hold up to the elements. I soldered and used electrical grease in heat shrink tubing on mine. Check the grounds also. You need to inspect the whole system.