Aspects of the present invention are related to methods and systems for determining a background color in a digital image. A background color value for a pixel in an image may be determined by summarizing the color values in a color buffer, wherein the color values in the buffer may have been selectively added to the color buffer, from the image data, based on one, or more, criteria which may be related to edge density, image uniformity, non-local color information, a foreground color estimate and other selection criterion. The background color for the pixel maybe determined in a forward scan and updated during a backward scan.