A distance measurement device includes a detection unit, an optical path forming unit, a first reduction unit that reduces, based on a detection result of the detection unit, influence of variation of the optical axis of the image formation optical system, a second reduction unit that is disposed in a different part from the common optical path and reduces variation of the optical axis of the directional light based on the detection result of the detection unit, and a control unit that, in the case of operating the first reduction unit and the second reduction unit at the same time, controls the first reduction unit and the second reduction unit to reduce variation of an irradiation position of the directional light in the subject image received as light by the light receiving section.