It may help to think about the difference between sound absorption and deadening, which are not (generally) the same thing. Dynamat and similar products work not by absorbing sound but by adding mass and possibly stiffness to the surface, so it absorbs and vibrates and transmits sound less. It's blocking sound, but not absorbing it to any great degree. Acoustic foam, on the other hand, absorbs the sound; it damps out the vibrations in the air.
If your noise is being transmitted mainly the generator vibrating the structure of the RV and then it vibrating and making noise, then sound absorption materials around the generator aren't going to have much effect at all. Deadening the walls etc. that are vibrating will help, as would improving the mounting so less vibration is transmitted.
If you do put foam or other materials around the generator compartment, make sure they are noncombustible, can take the heat, and are compatible with gasoline and similar chemicals. Many acoustic foams and similar products would not be prudent to use in such an application. Others are fine, of course.
For the coset walls, if they're constructed like the interior walls in my RV, stiffening them by adding some more framework inside or building a sandwich with rigid foam board would help a whole lot. Mine are just thin luan or something similar attached to studding, which can act something like a drum head or flapping tarp.