A semiconductor device and a method of manufacturing a semiconductor device are disclosed. The method includes forming a trench, in a vertical direction of a semiconductor substrate having a plurality of photoelectric converting elements arranged on the semiconductor device, at positions between the photoelectric converting elements that are next to each other, forming a first conductive-material layer in and above the trench by implanting a first conductive material into the trench after an oxide film is formed on an inner wall of the trench, forming a first conductor by removing the first conductive-material layer excluding a first conductive portion of the first conductive-material layer implanted into the trench, and forming an upper gate electrode above the first conductor, the upper gate electrode configured to be conductive with the first conductor. The semiconductor device includes a semiconductor substrate, an image sensor, a trench, a first conductor, and an upper gate electrode.