A method beginning at (10) is provided for creating high-fidelity visible coloring from infrared images of a scene under surveillance. The infrared images captured at (12) are analyzed at (14) to determine if an object, such as a face, is identifiable within the image. If an object is identifiable at (16) the object features are compared to a plurality of stored object features at (20). If there is a match at (22), the color characteristics of the object are obtained at (24) and the object is colored at (26) based on the stored database feature information. If there is no match at (22) or identifiable object at (16) and object color cannot be identified at (23), the image is analyzed at (28) to determine if a pattern, such as clothing, is identifiable within the image. If a pattern is identifiable at (30), the color characteristics of the pattern are obtained at (34) and the pattern is colored at (36) according to infrared reflectance characterization in conjunction with the stored pattern information. If no pattern is identifiable at (30), the non-pattern and non-feature containing portions of the image are colored at (38) according to infrared reflectance characterization.