A focus detection unit to adjust a focal point of an image, of an object, formed by an optical system includes a first output section, a second output section, and a projection optical system. The first output section includes a first light modulation element configured to generate a first pattern image based on incident light and is configured to output the generated first pattern image. The second output section includes a second light modulation element configured to generate a second pattern image based on incident light and is configured to output the generated second pattern image. The projection optical system is configured to project the output first pattern image and the output second pattern image such that the output first pattern image and the output second pattern image have a predetermined positional relationship at an in-focus position of the optical system.