I live in Bakersfield, California and there are a lot of Valley Fever cases here. I have had two miniature schnauzers for the last ten years and they are both always exposed to the same environment. One of them came down with Valley Fever a couple of years ago and I almost lost her. It cost a small fortune and a lot of TLC, but she is still with us and doing just fine now. My other little guy never suffered a single symptom. I don't know why it worked out that way, they were/are both healthy dogs, but there you have it.
My feeling is, don't live your life avoiding what you fear could happen.