A method of manufacturing a semiconductor device includes: loading a substrate into a process container after dry-etching a portion of a silicon film formed in a recess on the substrate; performing etching to partially or entirely remove the silicon film remaining on a side wall inside the recess by supplying an etching gas selected from a hydrogen bromide gas and a hydrogen iodide gas into the process container of a vacuum atmosphere while heating the substrate; subsequently forming a silicon film inside the recess; and heating the substrate to increase a grain size of the silicon film.