You are not crazy. 
Just traveling together and seeing things most people never get to see will create strong family bonds and memories that will last forever. For that reason alone you should do this!
That said, we've know a number of families that home schooled their children. There seem to be 2 types, those who do and succeed, those who do and fail. The failures are due to lack of dedication. They thought it was going to be easy. It's not. We considered it and realized it wasn't for us. It certainly seems like you are clearly thinking this out. I think you're going to be successful