To ultimately answer your original question:
NO. Absolutely not. Metal does not fatigue in a manner that requires it to need "rest" to "recover." Metal does not "heal."
If your hitch setup is working on Day 1, and you do not make any significant changes, it will work the same on Day 21.
That means, despite your repeated denials, SOMETHING is changing between Day 1 and Day 3 to cause the handling issues you are describing.
Just a fair warning, but people will soon grow frustrated with your repeated denials and dismissals of their suggestions. Either you have one of the many common problems plaguing trailer towers everywhere, or you have magical gremlins.
Somehow I place a much higher likelihood that it's a common problem, not magical gremlins.
It could still be driver fatigue. Even though you "rest" you are not 100% the next morning as you would be sleeping at home in your own bed. You may be 85% on Day 2 and only 70% on Day 3.