As a reference, when they talk about the winds of a hurricane, they are talking about the winds at 30 above ground level. In other words, other homes, trees, etc, all affect the surface winds which are always much lower than the winds 30 feet up. Of course if you live on the top of a hill, with no trees or other homes around you, you are going to experience higher winds speeds than a neighborhood may receive.
As a contractor, I built my current stick house to withstand Cat 5 winds at ground level, which is next to impossible to attain. By the same token, tornado winds, which often come with a hurricane, can bring high winds at ground level.
Flooding is always a problem from storm surge but depending on the drainage of your area, heavy rains can also cause flooding.
Jacksonville, NC is not very far inland, and may very well experience flooding in the low lying areas, so I would make good decisions on what is the best for you and your loved ones.