Optical measuring apparatus includes: a light source irradiating an object to be measured; a splitter splitting transmitted light or reflected light from the object to be measured; a phase changer changing a phase of a first light which is one of the lights split; a phase fixer maintaining a phase of a second light which is the other light split; a multiplexer multiplexing lights output from the phase changer and the phase fixer; a detector detecting the light (interference image) output from the multiplexer; and a controller that extracts a reference point from the interference image, when a displacement of the reference point is detected, corrects a luminance value for each pixel of the interference images in accordance with a displacement of the object to be measured indicated by a displacement of the reference point, constructs an interferogram based on the luminance value for each pixel of the interference images after the correction.