An image processing apparatus includes: an area extracting unit that extracts a candidate area of a classification target area in which a pixel value does not correspond to a three-dimensional shape of an imaging target based on pixel values of an intraluminal image acquired by imaging the inside of a lumen or information of a change in pixel values of peripheral pixels; and an area classifying unit that classifies the classification target area out of the candidate area based on the pixel values of the inside of the candidate area, a boundary portion of the candidate area, or a periphery portion of the candidate area.