A method and apparatus for processing a depth image determines a number of mods (NoM) for corresponding pixels in a plurality of depth images. The corresponding pixels may represent a same three-dimensional (3D) point. The NoM may be determined to be a value for minimizing a Markov random field (MRF) energy. A depth value for one depth image may be recovered, and a depth value for another depth image may be updated using the recovered depth value.