A hologram reproducing apparatus includes a first reference light path leading laser light from a laser light source so that a hologram recording medium is irradiated with first reference light; a diffracted light path leading diffracted light, generated from the hologram recording medium by irradiation with the first reference light, to a light receiving element having a plurality of pixels; and a second reference light path leading second reference light, having the same polarization direction as that of the diffracted light, from the laser light source to the light receiving element. The second reference light path is provided with a phase modulating element so that the phase difference between the diffracted light and the second reference light in a light receiving surface of the light receiving element is adjusted within a predetermined range.