Upon observation of multi-viewpoint images, even when the viewpoint moves continuously, the image cannot normally be displayed in correspondence with the continuous movement of the viewpoint, and an image from a viewpoint closest to the current viewpoint is displayed. To prevent this, each two adjacent images of the multi-viewpoint images are interpolated by detecting corresponding pixels and generating an interpolated image in accordance with the correspondences among pixels. The interpolated image is generated in such a manner that an epipolar plane image is generated by juxtaposing corresponding lines of the multi-viewpoint image, straight lines are detected from the epipolar plane image, and the intersecting points between interpolated lines and the detected straight lines are detected as interpolated pixels. In this case, as for a point for which a straight line cannot be detected, a straight line passing the point is estimated, thereby generating an interpolated pixel. When a plurality of straight lines are detected for a pixel of interest, the priority level is assigned to the pixel of interest in correspondence with the number of detected straight lines. When the priority level is smaller than that of another pixel through which the detected straight line passes, the straight line is ignored, and an interpolated pixel is not generated.