Provided is method and apparatus of compressing and restoring an image using filter information. The image compression apparatus may generate a reduced image by sampling an input image, determine filter information based on the input image and a decoded reduced image, and code the filter information and the decoded reduced image. Accordingly, a deterioration in an image quality may be prevented in transmitting the image using a limited bandwidth.