The confocal displacement sensor includes a light source for light projection configured to generate light having a plurality of wavelengths, a plurality of pinholes, an optical member configured to cause an axial chromatic aberration in the plurality of detection lights respectively emitted via the plurality of pinholes and converge the plurality of detection lights toward the measurement object, a spectroscope configured to respectively spectrally disperse, in the detection lights irradiated on the measurement object via the optical member, a plurality of detection lights respectively passed through the plurality of pinholes by being reflected while focusing on the measurement object and generate a plurality of light reception waveforms representing light reception intensities for each wavelength, and a measurement control section configured to statistically process the plurality of light reception waveforms and generate a representative light reception waveform from the plurality of light reception waveforms.