I once had a store where two weeks after delivery of the new model year vehicle, not one had been delivered to a customer. When I went to the Sales Manager, I found almost 100% of the inventory was on "hold" based on some conversation between a salesman and a previous customer. No deposits.
The dealer has money invested in each unit and no way to recover that money until it is paid for and delivered. To have any order in the process, the unit is either delivered, has a deposit on it with a scheduled delivery, or it is for sale. Otherwise, it is a very poorly run dealership. It is silly to think any business owner is willingly going to potentially tie up millions of dollars based on a perspective buyer having an honest face.
With all that said, in the case the OP describes, the dealer doesn't have a deposit. There is no signed contract, no exchange of merchandise, and no legal transaction has taken place in the statutes of any state. The dealer has nothing to lose by refusing to reverse the credit card deposit, other than credibility, and is simply bluffing. Dispute the charges with the credit card issuer and wait. The dealer was negligent in not obtaining a signed agreement and in this case will pay the price.