Color separation systems and methods improve color constancy and smoothness of a color lookup table (LUT) for a printer. A plurality of nodes of the LUT may be defined in colorimetric space, and the nodes out of the printer gamut may be mapped to the printer gamut surface. All possible colorant combinations are then determined that produce each node in the LUT in a device independent color space based on an inversion of the spectral based printer model. Next, a specific image quality metric combination based on a color inconstancy index (CII) and a gray component replacement (CGR) strategy is defined. And a colorant combination is determined for each and every node in the LUT based on this metric. A smoothing filter may be used to smooth the lookup table.