Sorry, can't offer any possible reasons for your tub and gray tank to fill. Has to be water going into the gray tank from some where. I seriously doubt if a small drip can fill a 40 gal tank plus tub in a few hours.
However, everyone has mentioned about turning the water supply off when you leave. The simple solution is to utilize your fresh water tank and pump. Fill the tank, turn off the water supply, and then use your pump whenever you need fresh water. Very simple.
Depending upon how much water you normally use will determine how often you have to fill your fresh water tank. We only turn on the pump whenever we actually need fresh water....we do not leave it on continuously. It doesn't take long for you to get into the habit of turning the pump on and off whenever you need it. Of course it helps to have multiple pump switches located through out the rv...or it's not that difficult to install them.
Ron