A diaphragm having an aperture, the size of which is equal to or smaller than .lambda.f/p (.lambda. is the wavelength, p is the imaging resolution, and f is the focal length of an imaging optical system) is arranged on the focal plane of the object space of the imaging optical system, and object light via the aperture is imaged by the imaging optical system. The object light via the imaging optical system and reference light are brought to interference to form interference fringes, and an image of the interference fringes is sensed. Using the hologram sensed in this manner, the image of the object to be sensed is reconstructed by adopting an imaging optical system equivalent to that upon imaging, and setting the positional relationship between the formed hologram and the imaging optical system in correspondence with that between the imaging optical system and the imaging surface upon imaging. In this way, a holography imaging apparatus which can vary the field of view and can sense a high-quality hologram using an imaging device having a relatively low spatial resolution is realized, and a holography display apparatus which can display an image by eliminating distortion with respect to the original image from a hologram sensed by the holography imaging apparatus of this invention is realized.