That's something that I have personally never experienced.
While working in the automotive field the only time I saw brake fluid that was contaminated was when someone added something to the master cylinder other than what was recommended by the manufacture.
I'm not saying don't flush it, I just don't see it as a problem if you don't.