Disclosed is a method and apparatus for restoring an image. The method and apparatus may detect boundary information associated with a boundary in an image, generate a reproducing kernel used to restore a hole in the image based on the detected boundary information, estimate hole information using the generated reproducing kernel, and restore the hole based on the estimated hole information.