Disclosed is a method of manufacturing an optical element capable of increasing surface accuracy of an optical surface of an optical element with a plastic lens portion. A cavity which is formed when a pair of molds are closed has a compression molding space and an annular injection molding space, and the method includes a compression molding step of casting a compression molding material into the compression molding space of the molds and an injection molding step of injecting an injection molding material into the injection molding space in a state where the molds are closed and forming an injection molded portion at the outer circumferential edge of the compression molded compression molding material. In the injection molding step, before the compression molding material reaches a temperature lower than the glass transition temperature, the injection of the injection molding material starts.