The low sulfur crude oil costs more to bring into the USA (both in dollars and the lives of American soldiers and the people in the countries where they operate). Higher input costs translates into higher prices. There are also fewer refineries processing the oil and with less competition there is more price gouging. I see this in California where there are now only two companies operating refineries in the entire state. 
Fuel taxes play a small part as well. Fuel taxes vary widely by state and different tax rates are applied to gasoline and diesel. The fuel taxes pay for the roads the trucking industry relies on so it is considered a necessary subsidy. With more economical cars and trucks being sold and bought the same amount of tax revenue needs to be collected from fewer gallons of fuel being sold so the rate per gallon will need to increase. 
Fuel is still comparatively cheap in the USA. When I see people driving down the highway at 75 MPH in their large SUV or truck to go 10 miles to the store to get groceries I have to figure gas is cheap compared to the value of the 2 minutes they are saving by driving at that speed.