No your not a Nutcase, your a very special person for taking that sweet little guy into your home. God Bless You! I think you will be surprised how that little guy reacts to a loving family.
I can attest to that with our little Boston Terrier that we adopted going on 3 years ago. He was in a very similiar situation and so we were told was terrified of men. I know some dogs from that situation can be a challenge but the rewards they give far out weigh the challenges. Trust me i know, our little guy has really wormed his little heart into our dog poor family of 5 and is a member of our family. Good luck and Thank You for fostering that little guy. I bet you end up adopting him and giving him a forever home before you know it.