An image processing apparatus includes a depth image obtaining unit configured to obtain a depth image including information on distances from an image-capturing position to a subject in a two-dimensional image to be captured; a local tip portion detection unit configured to detect a portion of the subject at a depth and a position close from the image-capturing position as a local tip portion; a projecting portion detection unit configured to detect, in a case where, when each of the blocks is set as a block of interest, the local tip portion of the block of interest in an area formed of the plurality of blocks adjacent to the block of interest, becomes a local tip portion closest from the image-capturing position, the local tip portion as a projecting portion; and a tracking unit configured to continuously track the position of the projecting portion.