A method of the disclosure includes receiving, by a processing device, a document image, dividing the document image into a plurality of patches and determining, for each patch, whether the patch is monochromatic or polychromatic. It further includes clusterizing a plurality of monochromatic patches into a plurality of clusters within a color space, wherein each cluster corresponds to a color layer of a plurality of color layers of the document image, and segmenting each polychromatic patch into a corresponding plurality of monochromatic segments. The method also includes, for each polychromatic patch, associating each monochromatic segment of the corresponding plurality of monochromatic segments with a cluster of the plurality of clusters, and utilizing the plurality of clusters for performing an information extraction task on the document image.