The pixel characteristics are prevented from being degraded due to diffusion of electrons and Fe (iron) from the surface of an element isolation trench formed in the top surface of a semiconductor substrate into a photodiode forming the pixel of an image sensing element. Further, oxygen is prevented from being diffused from a boron oxide film formed at the surface of the element isolation trench into the photodiode. In the top surface of the semiconductor substrate, a trench for embedding an element isolation region surrounding a photodiode-forming region is formed. Then, B (boron) is doped into the surface of the trench to form a semiconductor layer. Subsequently, the boron oxide film resulting from the reaction between the boron deposited at the surface and oxygen is removed by APM washing. Then, a heat treatment is performed to diffuse the boron in the semiconductor layer.