Provided are a hologram display device and a method of manufacturing the hologram display device. The hologram display device includes a light source unit that emits light, a spatial light modulator that modulates the light emitted from the light source unit, and a random pinhole panel. The random pinhole panel includes a plurality of pinholes of a random position or a random size and is arranged in line with an output part of the spatial light modulator. In the hologram display device and the method of manufacturing the hologram display device, a position and size of a random pinhole on the random pinhole are not limited to inside each pixel of the spatial light modulator.