eraser marks on Travel Trailer

Last weekend when we returned home from our Northern Michigan trip I noticed silver metallic specs all over the road side of the trailer and front cap just on the road side, well I had to use magic eraser and black streak remover to get if off which it did on the side but the front cap I made the specs smaller but could not remove them. But that is the least of my troubles when the TT dried I noticed there were eraser marks where I really had to scrub to get the specs off I can post pics later but if I use rubbing compound will it take the marks off it is a smooth sided TT 2003 Aruba I tried washing it with the liquid blue coral soap and wax combo but the marks are still there.

Thanks,