That is just what my wife and I do, sort of. We do not travel just to fish and hunt, but rather do them during our travels. But the down side is just as JoeGood988 stated.
For example and by coincidence, we just returned after spending two months in South Dakota. Had a great time fishing and stayed long enough to get in ten days of SD's Ringneck Pheasant hunting. Expensive for non-residents is right. $110.00 a peace for a 10 day small game hunting license. An annual small game hunting license is not available in SD for non-residents. But it was worth every dollar. We had a wonderful time and great success hunting SD's wild birds on public land. :)
Big game licenses and permits in various states are a whole different matter and are even much more expensive and extremely hard to obtain in many states for non-residents.