A method includes forming image sensors in a semiconductor substrate, thinning the semiconductor substrate from a backside of the semiconductor substrate, forming a dielectric layer on the backside of the semiconductor substrate, and forming a polymer grid on the backside of the semiconductor substrate. The polymer grid has a first refractivity value. The method further includes forming color filters in the polymer grid, wherein the color filters has a second refractivity value higher than the first refractivity value, and forming micro-lenses on the color filters.