There is provided a method of fabricating a semiconductor device having plural light receiving elements, and having an amplifying element, the method including: a) forming an active region on the semiconductor substrate for configuring the amplifying element; b) forming a light receiving element region on the semiconductor substrate for forming the plural light receiving elements, with the active region acting as a reference for positioning; c) implanting an impurity into the light receiving element region; d) repeating the process b) and the process c) a number of times that equals a number of diffusion layers in the light receiving element region; e) after implanting the impurity, performing a drive-in process to carry out drive in of the semiconductor substrate; and f) the process e), forming an amplifying element forming process by implanting an impurity in the active region.