The number of masks is reduced in a method of manufacturing a semiconductor device that has a transistor and a photoelectric conversion element on an insulating surface. In a manufacturing method of the present invention, semiconductor layers functioning as a source region, a drain region, and a channel formation region of a transistor are formed at the same time an n type semiconductor layer and p type semiconductor layer of a photoelectric conversion element are formed. Connection wiring lines to be electrically connected to the n type semiconductor layer and p type semiconductor layer of the photoelectric conversion element are formed at the same time a source wiring line and a drain wiring line of a transistor are formed. In a doping step using an impurity element that gives one conductivity type, a semiconductor layer of an n-channel transistor and the n type semiconductor layer of the photoelectric conversion element are simultaneously doped with the impurity element and a semiconductor layer of a p-channel transistor and the p type semiconductor layer of the photoelectric conversion element are simultaneously doped with the impurity element.