3-4 cents per mile? Not worth it.
But if I fill up to the top before I pull out of town and come back on 1/4 tank, I know I pretty much go the most bang for my buck fuel-wise. I fill up on-base with the Military Star Card. Today, diesel is $3.629, or $3.579 with the Star Card. The local truck stops are $3.789. When I am purchasing 85 gallons to start the trip, saving 21 cents per gallon adds up to nearly $18 saved on that tank.
There's also something to be said for common sense. If I were taking a trip from Kalispell, MT to Banff and back, I would fill up in Whitefish, MT before crossing the border. Its currently $4.059, yet its north of $5.00 up there, a savings of about $1.20/gallon (about $120 less per tank).