A semiconductor device includes a substrate, a device layer, a composite grid structure, a passivation layer and color filters. The device layer overlies the substrate. The composite grid structure overlies the device layer. The composite grid structure includes cavities passing through the composite grid structure, and the composite grid structure includes a metal grid layer and a dielectric grid layer stacked on the metal grid layer. The passivation layer conformally covers the composite grid structure. The color filters respectively fill the cavities.