I would not call it horrible. But I would question the wisdom of it. Are you trying to get out from under house payments to improve cash flow? A "nice house in a nice neighborhood" is almost surely going to appreciate in the coming years. An RV is destined to depreciate. What if you decided after 6 (or 8, or 10, or ?) years you wanted to be back in a house? If you had invested the results of the improved cash flow you would probably have adequate money to get back into a house without hurting too much. If the money was spent for other things, it would make that a more difficult task.
As far as child(ren), if they grew up in an RV I don't think they would have a problem as they grew older, at least in the earlier years. Although I am sure most teenagers would prefer to have their own room with some privacy, rather than a small bunkhouse room.
But in the end, only you and your family can decide what is right for you.