An image processing device includes: a specific candidate area extracting unit configured to extract a specific candidate area that satisfies a predetermined condition from an intraluminal image captured inside a body lumen; a reference area setting unit configured to set a reference area that includes at least a part of the specific candidate area; a local area extracting unit configured to extract local areas based on the reference area; a local feature data calculator configured to calculate local feature data that is feature data of each of the local areas; a weight setting unit configured to set a weight depending on each of the local areas based on the specific candidate area; and a feature data integrating unit configured to integrate the local feature data.