A method for segmenting an image includes extracting unary potentials for pixels of the input image. These can be based for each of a set of possible labels, on information for a first channel in the image, such as in the visible range of the spectrum. Pairwise potentials are extracted for neighboring pairs of pixels of the image. These can be based on information for a second channel in the image, such as in the infrared range of the spectrum. An objective function is optimized over pixels of the input image to identify labels for the pixels. The objective function is based on a combination of ones of the extracted unary and pairwise potentials. The image is then segmented, based on the identified pixel labels. The method and system can provide an improvement in segmentation over methods which use only the visible information.