Provided is an image processing apparatus and method for adjusting a color image using a depth image. The image processing apparatus may include a color image segmenting unit to segment a color image into a plurality of color segments, using a color layer of the color image, a depth image segmenting unit to segment a depth image into a plurality of depth segments, using a depth layer of the depth image, and a layer boundary adjusting unit to adjust a boundary value of the color layer, using the plurality of color segments and the plurality of depth segments.