For my wife and I, it's not so much when I should give up RV driving all together, but under what conditions should I not drive. We are lucky - she is much younger and a very good driver, so we always have a choice who drives.
My worst is in the rain at night; I just don't see the road well then, so it is always she under those conditions. My endurance is not what it used to be. I used to be good for all day driving, with suitable breaks. Now I rarely put in more than three hours a day, and that is in several stints with the wife driving shifts in between.
So for our family, the question is not yet about giving up the class A driving, it is about conditions and guidelines.