An apparatus and method for modifying an arrangement of colors of an input image to satisfy a user preference in consideration of color composition of the input image, the apparatus including: an extraction unit extracting one or more dominant colors from an input image; a center-of-gravity calculation unit calculating a first position of a center of gravity based on a position of each dominant color in a color space; a parameter calculation unit calculating a parameter for modifying the color arrangement of the input image based on the first position and a second position to which the center of gravity is moved from the first position according to a predetermined standard; and a control unit modifying the color arrangement of the input image according to the parameter.