Depending on the make of fridge (in my case a Dometic), mine was also noisy for a couple reasons...
One.. I was a cheap fan. There are quite an array of fans available and Dometic uses the cheapest, sleeve bearing fan available. I would suggest replacing it (as I did) with a FDB (Fluid Dynamic Bearing) fan with a computer designed blade. The FDB fans are silent in operation and a computer designed blade will reduce air flow noise. I picked up a couple on Flea-Bay for 10 bucks each, they make no noise whatsoever. But..
Two: You need to pay attention to how the fan(s) are mounted. Just like a FW pump thats mounted solidly to the substructure of the camper, the fan will transmit operational noise and the camper fram will amplify it. You need to isolate the fan from transmitting operational noise/vibration to the camper itself.
I did that by adding rubber grommets to the fan mounts. While it don't dampen everything. between the FDB fan itself and the rubber grommets in the mount, my fan noise is very minimal.
Personally, I prefer not to hear that noise when I'm sleeping at night. During the day not a big issue but at night, I want quiet. Before the modification (new fan(s) and mounts, I was turning off the fridge at night so I didn't have to listen to the fans.
Pretty cheap fix really, just takes some time on your part.