A method and apparatus for processing a multi-view image is provided. The method includes: extracting disparity information between an image of a first point of view and an image of a second point of view; and estimating a motion between two sequential images of the first point of view or the second point of view using the extracted disparity information. The apparatus may include a processor which is configured to extract disparity information between an image of a first point of view and an image of a second point of view in the multi-view image and is further configured to estimate a motion using the extracted disparity information between two sequential images of the first point of view or the second point of view.