A method and apparatus for processing hologram image data capable of optimizing image quality of a hologram image are provided. The image processing method includes receiving input image data, reading a header included at a predetermined location in the input image data, and generating hologram data configured to display a hologram image by performing a Fourier calculation and pixel encoding on the input image data based on at least one parameter recorded in the header, wherein the at least one parameter recorded in the header includes at least one of depth information, scale information, and gamma information.