A method is provided for generating a visualization for explaining a behavior of a machine learning (ML) model, the method includes inputting an image into a machine learning (ML) model for an inference operation. A first heatmap is generated for the image using a first visualization method. An area of highest attention is selected on the first heatmap based a predetermined threshold. The selected area is cropped from the image. The cropped selected area is upscaled. A second heatmap is generated for the cropped and upscaled selected area of the image. A final visualization is presented for analysis. In another embodiment, a computer program comprising instructions for executing the method is provided.