If you're saying yes to the customer who prefers Amex, you can have many more opportunities to earn back those fees and more. If you say no, you may have a one-and-done customer (or even one that walks away without completing the transaction).
I don't consider it anti-customer in the least. It's a wise business decision-why should those costs be passed to other customers when YOU have other options? Most people carry multiple credit cards and I don't think I know anyone who would just walk away instead of using their VISA or M/C. Those that do IMHO are somewhat selfish and may be problem customers in the long run.