I would highly recommend removing the windows and replacing whatever sealing they used with butyl tape. This will get you much a better seal than just a bead of caulk around the outside (which you can add anyway, if you want), and also it is a great opportunity to check for water damage.
On my rig I took the windows out and found that 2-3 had been leaking since they were installed because Coachmen didn't install the foam rubber tape correctly. After replacing the window a pizza cutter works great to slice off the extra butyl. You can even run the pizza cutter at an angle to bevel the butyl to shed water better.