Some users suggested that a solar panel cannot drain a battery very quick.
In fact, they did.
I installed a new diode and this fixed all issues. Apparently my old diode went bad and power was able to go back into the solar panel at night, so much that the batteries went dead fast.
Keep spare diodes on hand, they are easy to install.
I had got bad advice that the solar panel was not big enough to drain a fully charged battery over a few days. Solar panels will do this. Forget science that someone tells you. The reality is in the facts. A man with experience is more than a man with an idea.