We understand what you're saying. The wife and I both work 50-60 hours a week and when the kids were young (Now 17 and 19) we had to schedule out our time for trips. We took many 4-5 day trips and would pull the kids out of school on a Friday if needed.
Time is too short, your child will be grown and gone before you know it. MAKE THE TIME as you cannot get it back.
I now go on 1 or 2 trips to the TNF by myself each year. Our son will join us when he can and we now take 1 trip to the Smokeys without the FW as we like to stay right in Gatlinburg.
Best of luck, it's a hard decision. Get your wife and son involved in the decision.