Those who are giving you grief probably do not mean to be intruding on your privacy, but ARE trying to protect handicappers rights. It is unfortunate that they verbally abuse you.
I've seen people without visible signs of a handicap & people who are cheating. I don't speak to either. I have called the police on one who was obviously a cheat, going into the gym, lifting more weight than I could & running for 1/2 hour.
Sorry that people harass you while trying to protect handicappers rights. They should realize some disabilities are not visible. Try not to take it personal.