The problem is that there is so much fraud with the disabled permits. I've seen television reporters questioning people about their permits, only to find out they were using someone else's permit.
Now, in some states and provinces, all a person needs to do is convince a doctor that they have a disability which will allow them to get a permit from the motor vehicle department. There are no checks and balances, and there is very little in the way of follow up years down the road. I know a person who got a permit for a sore knee. A few years later, it got bad enough for them to have the knee replaced. Did they give up the permit when they healed up? Heck no. It's nice to be able to park closer. In fact, he'd carry a cane in order to keep from getting bothered.
So if you don't have an obvious disability, you will get harrassed and yelled at. It's just a fact of life. Too many people are taking advantage of a system that was designed to help severly handicapped people live life a little easier. I don't have much sympathy for those catch some flack when they don't really have the special needs.