A method for separating foreground and background contents in a document image is provided. The method first computes a pixel-wise map of maximal local features (e.g., local variance, local contrast, etc.), which is binarized to generate a mask for potential foreground. In order to utilize color information effectively, the local feature map is computed using all color channels of the image. Then the background image is obtained by inpainting the mask regions from the non-mask regions of the original document image. Adaptive thresholding is applied to the difference between the original document image and the background image to obtain the binary foreground image. Post-processing of the binary foreground image can further remove undesirable elements. Finally, a more accurate background image can be obtained by inpainting the original document image using the binary foreground image as a mask.