I do mine every fey years if I remember. To me these are no different than cars of old where you had to pack the front wheel bearings from time to time.
The fact that you are pumping 4 or 5 pumps of grease into the zerk every year is almost as ineffective as not pumping at all. But the fact that you have not had a failure is actually a testament to not really need to pack every year. If the bearings are properly greased and adjusted then they will go many years without care.
Yes you will read reports of inadequately lubricated bearings on here. But I think most people don't really realize what a properly packed assembly looks like. I think they are expecting a bunch of grease in the hub between the inner and outer bearings. But this is an incorrect assumption. A properly packed assembly is grease in the bearing and little to no grease in the hub itself.
Below is Dexter Axles steps for packing a bearing. In the steps there is no mention of packing anything except the actual bearing. I personally disagree with the 12 month 12,000 mile statement as a manufacturers CYA. But I left it on here as it is Dexters procedure not mine.

Click For Full-Size Image.