An image processing apparatus for processing spectral image data of an object includes a normalization unit configured to normalize spectral data in each pixel of the spectral image data with respect to luminance level, a deviation amount calculation unit configured to calculate a deviation amount between the spectral data of a reference pixel and the spectral data of a target pixel to be clustered, which are normalized by the normalization unit, among each of the pixels of the spectral image data, and a segment determination unit configured to determine an affiliated segment of target pixel in the object of the spectral image data based on the deviation amount calculated by the deviation amount calculation unit.