It really depends on too many variables. From a personal point of view I would do the following.
If you have a huge amount of liquid funds doing nothing, pay it off. If paying the mortgage off would deplete your saving, leaving nothing for future emergencies or opportunities that might arise, don't.
In the middle or in the position that paying it off leaves you short of liquidity, split the difference and refinance. Currently, 15 year fixed rates are under 3%. Rates today are near historic lows. While history does not drive the future, it is usually a pretty good predictor. Personally, I believe you would wake up in 2021 and pat yourself on the back for having locked in a loan at 2.9%.