A method, system and computer program product for progressively encoding a digitized color image with M distinct colors by assigning each of the M distinct colors to an associated subset of pixels in the image, is provided. This involves: (a) initializing a tree structure with at least one starting leaf node comprising a subset of the M distinct colors; (b) determining at least one representative color for each starting leaf node; and (c) growing the tree structure by (i) selecting a leaf node n to become a non-leaf node based on the combined distortion and entropy rate resulting from turning the leaf node into the non-leaf node; (ii) allocating each color in leaf node n to one of the two new leaf nodes; (iii) determining a representative color for each new leaf node; and (iv) encoding the resulting index information, representative color information, and pixel information.