A method of processing an image includes obtaining a position of a viewpoint corresponding to the image, selecting, based on the position, two reference rays closest to a ray included in the image from among a plurality of reference rays included in reference images photographed in advance from different reference viewpoints; determining whether a difference between color values of the selected reference rays is less than or equal to a threshold; and determining a color value of the ray based on the color values of the selected reference rays in response to the difference between the color values of the selected reference rays being determined to be less than or equal to the threshold.