If there is no breeze, it's very much a concern, particularly for ground level exhaust but even for roof level or portable generators, in the right conditions, it can be a concern. The further the exhaust is away from the rig, the better.
RV's are not sealed up very well so if the exhaust hangs around, it likely will find it's way in. We usually set up our portable generator 15-20ft from the rear of our trailer (bedroom is in the front) with the exhaust pointed away.
Events can be problematic as they often pack the rigs in close and lots of people are running generators, so it may not even be your generator that causes the problem.