An image processing apparatus and method for processing an image using a multi-view image and pixel information are provided. The image processing method includes determining a view image that corresponds to a predetermined display direction of a display pixel of a display, determining at least one view image pixel corresponding to the display pixel in the determined view image based on a position of the display pixel in the display and pixel position information of the determined view image, and determining a pixel value of the display pixel based on a pixel value of the determined at least one view image pixel.