Grey tacky tape is the best to use to go between the window frame and the coaches' outside wall.
If you have radius windows with the screws on the inside of the rig, remove them leaving one loose until you are ready to remove the window. When you have a helper on the outside, push hard around the edge and frame of the window from the inside. It may take some effort to break the seal loose.
Once window has been removed, clean the area around the outside. Install new tacky tape and then window. Once all screws are tight, excess putty tape has been trimmed off and you are happy, apply some sealant around the top and down side of the window.