You are correct. Invoice price is not what the dealer pays. They get other incentives for selling particular models and options or meeting a particular volume or quantity of units sold.
You should not have have pay more than invoice price (which includes delivery fee) plus tax and license. If there are any rebates or special financing options, these are additional discounts to the above. Two of my vehicles were purchased at invoice price minus rebates and one was purchased for $500 under invoice minus rebates. Taxes and licensing obviously are not negotiable, but the amount you you negotiate will be taxed (sometimes before rebates). These were custom orders and not off their lot. I would expect even greater discounts on stock that has been sitting or been test driven by others.