I drive by the feel of the road as you said and visibility. I've also found that very strong driving lights help in heavy downpours as they give you better visibility and therefore confidence in what you're doing. I can't give you a speed because each situation is different.
And speaking of wind gusts, last year I had a gust hit our DP in the side up in Montana someplace. It was so strong that it actually moved the Velvac mirror out of adjustment. I had to get out the allen wrenches, go outside and adjust it.
Bill