At what point do you call it quits on repairs??

Anytime silicone sealant (except for $$ absolutely pure silicone) is in daylight, it is deteriorating. Nothing will adhere to it, even more new silicone or paint. That's why it is a no-no for using on an RV.
There are a variety of polymer sealants at the same price point so using them is where you want to start. I found that Geocel Construction sealant is very durable and is paintable.