A method and apparatus for processing a three-dimensional image are provided. The method includes receiving original color data and original depth data of a plurality of layers of an original holographic image, selecting reference layers from among the plurality of layers, mapping adjustment color data of a non-selected layer, which is determined based on using the original depth data of the non-selected layer and the reference layers, to each of the reference layers, and generating a computer generated hologram image by using the original color data of the reference layers and the adjustment color data that has been mapped to the reference layers.