A semiconductor device is provided. The semiconductor device includes a semiconductor substrate. An isolation feature is disposed in the semiconductor substrate to define a pixel region and a periphery region of the semiconductor substrate. A transistor gate is formed on the semiconductor substrate in the pixel region, in which the transistor gate has a first sidewall and a second sidewall opposite to the first sidewall. A photodiode is disposed in the semiconductor substrate and adjacent to the second sidewall of the transistor gate. A patterned spacer layer is formed on the photodiode and on the transistor gate. The patterned spacer layer includes a first sidewall spacer on the first sidewall of the transistor gate, and a protective structure covering the photodiode and a top surface of the transistor gate.