Your anode rod did exactly what it was suppose to do ... deteriorate! The white particles are the residue from the rod. It's designed to do that. You do need to flush out your water heater every now and then for this very reason.
About water softeners. We have one at the house and although we are on a well, and do not have chlorine, all it does is, basically, add salt and somehow soften the water. It does nothing for iron (which we have strong iron in our home water.
Actually, the city chlorine is good for your water system. It helps to keep your system sanitized, and you don't have to do anything. Because we are on a well, if we go camping and have advantage of using chlorinated water, we fill the fresh water, run it through the lines and let it self-sanatize everything for us. When we lived it town, we didn't have to worry about that, because we kept city (chlorinated) water in the tank all the time.
Another thing about water softeners. Although they do help remove some chemicals, the salt that is replaced is not good for drinking. That's why in homes, usually the kitchen faucet cold water is never hooked up through the water softener, but straight from the water source, be it city or a well.
