An image processing apparatus includes an input unit for inputting image data, an attribute information generation unit for generating attribute information about each pixel of the image data, a block division unit for dividing the image data into a plurality of blocks of a predetermined size, a histogram generation unit for generating a histogram of colors and the number of pixels of each color existing in a focused block divided by the division unit, and a color replacement unit, if it is not determined that the area having the focused color is a significant area, for replacing a color of a pixel existing within the area having the focused color with a different color so that the color of the area having the focused color becomes the same as the color of another area contacting the area having the focused color and having the different color according to a predetermined condition.