In this video is the only thing that has come close to describing what I've seen in the same metal water cells. They say that the water forums a crystal at the top and they could get a voltage from it. This would explain why having one plate barely touching the water at the top gave a lot of voltage increase. Its a long video but describes a lot and is good information.
So it was the crystals in the water that I was tapping, like a crystal battery or Petrovoltaic battery.