So I went into an RV parts dept today and bought the two plastic window frames (found some cracking in the old ones) and was helped by a, seemingly, knowledgeable guy.
Asked him what is used for the seals and he directed me to butyl tape (for the outside of the frame(s) between the door and frames) and some silicone for the interior of the frames (to seal the actual glass). The tape was about 3/4 in wide, but he said to just apply and trim the excess.
Took the items home and spent the next couple hours cleaning the old stuff off the door, applying the tape (per the instruction sheet that accompanied the frames) and silicone. Then fiddled quite some time getting the frames in the door, without the frames or glass falling out (solo job, would have been much better with a second pair of hands).
And, found out it all would not fit because the butyl tape was too thick and left way too big a space between the interior frame and glass, and there was absolutely no way to compress it enough to get the frames to join to insert the screws.
By the time I pulled it all apart (everything was sticking together from trying to put it together), the silicone was a mess and the tape was now a big ball of sticky, so had to throw it out. So much for the employees advice.