A method includes forming a plurality of image sensors on a front side of a semiconductor substrate, and forming a dielectric layer on a backside of the semiconductor substrate. The dielectric layer is over the semiconductor substrate. The dielectric layer is patterned into a plurality of grid-filling regions, wherein each of the plurality of grid-filling regions overlaps one of the plurality of image sensors. A metal layer is formed on top surfaces and sidewalls of the plurality of grid-filling regions. The metal layer is etched to remove horizontal portions of the metal layer, wherein vertical portions of the metal layer remain after the step of etching to form a metal grid. A transparent material is filled into grid openings of the metal grid.