The confocal displacement sensor includes a first optical fiber, to a first incident end of which light for light projection is input, the first optical fiber outputting the light from a first emission end, a second optical fiber, a second incident end of which is disposed to be opposed to the first emission end, the second optical fiber emitting light input via the second incident end from a second emission end, an optical member configured to cause an axial chromatic aberration on detection light emitted via the second emission end and converge the detection light toward the measurement object, a fiber connecting section configured to detachably connect the second incident end to the first emission end, and a refractive index matching material disposed between the first emission end andthe second incident end.