If it's the check valve in the pump, it will also cause the pump to cycle briefly every once and again if it's left on with no water being used and no city water connected. (Pressure leaks down through the check valve, causing the pump to cycle until the pressure is restored.)
If you put in an external check valve, put it in the supply side to the pump, not the output. If it's at the output the pump will still cycle periodically when there's no demand. It may also be a little bit of crud or something stuck in the pump valve, and cleaning (or just using) the pump may fix the problem.
As rv2go suggests, it could also be the fill valve not closing completely for whatever reason.