A method and apparatus for processing a multi-view image are provided. A priority may be assigned to each hole pixel in a hole region generated when an output view is generated. The priority of each hole pixel may be generated by combining a structure priority, a confidence priority, and a disparity priority. Hole rendering may be applied to a target patch including a hole pixel having a highest priority. The hole pixel may be restored by searching for a source patch most similar to a background of the target patch, and copying a pixel in the found source patch into a hole pixel of the target patch.