An image sensor with stress adjusting layers and a method of fabrication the image sensor are disclosed. The image sensor includes a substrate with a front side surface and a back side surface opposite to the front side surface, an anti-reflective coating (ARC) layer disposed on the back side surface of the substrate, a dielectric layer disposed on the ARC layer, a metal layer disposed on the dielectric layer, and a stress adjusting layer disposed on the metal layer. The stress adjusting layer includes a silicon-rich oxide layer. The concentration profiles of silicon and oxygen atoms in the stress adjusting layer are non-overlapping and different from each other. The image sensor further includes oxide grid structure disposed on the stress adjusting layer.