The disclosure herein generally relates to image processing, and, more particularly, to a method and system for impurity detection using multi-modal image processing. This system uses a combination of polarization data, and at least one of a depth data and an RGB image data to perform the impurity material detection. The system uses a graph fusion based approach while processing the captured images to detect presence of the impurity material, and accordingly alert the user.