Sounds like a great job, but the homelife will be stressful (I believe). Big difference between spending a couple of weeks a year, in small confines, and doing it every day for the rest of your life (or the foreseeable future). As someone pointed out, where is the 'break' for your wife? You will also be coming home every day to cramped living quarters etc...
Add to the equation the fact you will have to do more maintenance (way more in actuality) than you have to do for a stick and brick house. And you don't have the opportunity to decide to put it off til next week. And if something does break just before your week off, then you're not going anywhere.. you're staying put to fix whatever 'it is'.
If something breaks that is beyond your capability, there goes your living quarters.. off to the dealer or repair shop, and now you'll be looking for a hotel that allows three dogs.
Recipe for some unhappiness. But, if you can accommodate all that, and a bunch more unforeseen circumstances, then more power to you.
Sounds like you guys are a relatively young married couple, still establishing your lives, and your relationships, and bringing a new infant into the world. Don't add the additional stress that this will no doubt bring, to the situation. If you can't afford the house you have, downsize to a smaller, more affordable one, but don't head down this road.
Of course, in the end, only you can make the decision, but since you asked for opinions, yes you are crazy to do it.
Mike