Sounds like from your second post that you already thought it was in a bad neighborhood. Would you want your wife or daughter to leave an office and go with a man (a stranger) to an area where there was nobody else (in this bad neighborhood)? And after he's told no, he asks again? By asking again for her to leave the office and go with you to an area in a bad neighborhood where there is no one else around, she told you that you ask too many questions and to go elsewhere.
Maybe there's more to the story, but based on what you have said so far, I'm with her.