The world is made up of nesters and nomads. US culture and the government promote the nester mentality and vilify the nomad mentality. Consumerist culture practically requires nesting - buy a house, fill it with stuff, buy a bigger house to buy more stuff, etc.
We haven't really had anyone say we are crazy or stupid. Many people tell us they wish they could do the same but they can't afford to. We didn't spend the last 17 years "Keeping up with the Joneses" and I think that is really where the jealousy comes from. At 47, we are able to retire comfortably, without debt. Many of them need to work at least another 10 years to get out from under the stuff they had to have. We can work if we want to, not because we have to; it makes a huge difference in one's outlook.