A hologram reconstructing apparatus is disclosed. The apparatus includes: unnecessary light removing means for causing the imaging means to receive necessary light only included in the reconstruction light; driving means for operating the hologram recording medium or a reconstruction light optical system; light detecting means for detecting the state that reconstruction light is projected to the unnecessary light removing means of the light cut by the unnecessary light removing means; and control means for obtaining the displacement of the hologram recording medium based on the information on a specific part of the projection state information detected by the light detecting means and controlling the driving means to cancel the displacement and move the hologram recording medium or the optical system.