A hologram display device includes a light source unit, a light guide plate, a spatial light modulator, a sensing unit, and a light source driving unit. The light source unit includes a plurality of light sources and emits light when at least one of the plurality of light sources is turned on. The light guide plate converts the light emitted thereto from the light source unit to a planar light beam. The spatial light modulator spatially modulates the planar light beam to produce a hologram image. The sensing unit senses a position of a user watching the hologram image, and the light source driving unit turns on the at least one of the plurality of light sources, based on information on the position of the user obtained by the sensing unit.