As an avid hunter, I say ride away. Just stick to the main roads. To be prudent, put on an orange safety vest. When I'm hunting, I try to get as far from roads and people as possible.
Hunting accidents are extremely rare. In 2010, the most recent statistics, there were 10 hunting accidents in California, and two of them were fatal. None of the 10 were while hunting deer. In states like California, the hunter not only has to identify the game as deer, but must also count the number of points on the antlers in order to make a legal kill. This forces the hunter to take their time and make an accurate assessment of the target.
