An image sensor device includes a semiconductor substrate having a first side, and a trench isolation structure dividing the substrate into sensing units. Each sensing unit includes a first gate electrode and a second gate electrode disposed on the first side, and a first pixel and a second pixel extending into the substrate and disposed between the first and second gate electrodes from a top view perspective. The first pixel is disposed under the second pixel and electrically connected to the first gate electrode, and the second pixel is electrically connected to the second gate electrode. A method of manufacturing a semiconductor structure includes forming a trench isolation in a semiconductor substrate; forming a first pixel in the substrate; forming a second pixel in the substrate over the first pixel; forming a first gate structure over the substrate; and forming a second gate structure over the second pixel.