Help identify purpose for holes in window frame

Bottom line: Is there any reason why I shouldn't caulk these holes in my window frame that lead directly to the inside?

You can see from the photos the holes are both top and bottom. But at the top the holes are covered by a rubber insert. The bottom ones are bare.