Homeless Cat -- Now Spotacus' Legacy

For the last month or a bit more, I have been feeding a young cat. It looked like a very old kitten or a cat just forming from a kitten. I have been calling it Mr. cat. Not even sure it is a male. He wont let anyone near him.If he sees me he runs away. I know he knows it is me feeding him as I have seen him hiding in the lilacs watching me put out his food. I began feeding him because I saw him eat 3 very dry and stale hamburger buns. I feel sure he wouldn't have eaten them unless he was very hungry. He was extremely thin but now, while not fat, is not skin and bones. I have made every effort to not upset him. Have never chased him, but he still avoids me. Does anyone have any ideas on how I can win his trust? I would like to be able to take him to the vet and have his shots and neutered. ( Maybe the neutering is why he avoids me) I was thnking of leaving the door open near where I feed him, but then I don't want any other wildlife to enter. Any ideas would be appreciated.