A method of forming a semiconductor device includes forming photodiodes extending from a front-side surface of a semiconductor layer into the semiconductor layer; forming transistors on the front-side surface of the semiconductor layer; forming an interconnect structure over the transistors, the interconnect structure comprising an inter-metal dielectric and metal lines in the inter-metal dielectric; etching first regions of a backside surface of the semiconductor layer to form trenches in the semiconductor layer and non-overlapping the photodiodes; after forming the trenches, etching second regions of the backside surface of the semiconductor layer to form pits in the semiconductor layer and overlapping the photodiodes; and depositing a dielectric material in the trenches and the pits.