A method of removing fog from the images/videos independent of the density or amount of the fog and free of user intervention and a system for carrying out such method of fog removal from images/videos are disclosed. The removal of fog from images and video involve airlight estimation and airlight map refinement based restoration of foggy images and videos. Advantageously, removal of fog from images and videos of this invention would require less execution time and yet achieve high perceptual image quality with reduced noise and enhanced contrast. The proposed method is adapted for RGB Color model and advantageously also for HSI color model involving reduced computational requirements and be user friendly and supposed to have wide application and use.