The images of an object taken at a plurality of viewpoint positions are input together with information indicating the plurality of viewpoint positions, and the viewpoint position of an observer is detected. One of the input images is selected on the basis of the detected viewpoint position of the observer, and the scan line of the image is determined in accordance with the viewpoint position of the observer and the viewpoint position upon taking the selected image, thereby generating an image from the viewpoint position of the observer. In this manner, distortion of the image to be generated is suppressed. On the other hand, input multi-viewpoint images are converted into light rays that travel in a space, and each light ray is stored in a light space corresponding to a plane, which is one of a plurality of radially arranged reference planes and has an angle the plane makes with the light ray, which angle is closest to right angles. On the basis of the image, pixel positions, and pixel values calculated from the stored light rays, a desired image is generated, thereby broadening the observation range within which movement of the viewpoint position is allowed, and suppressing the required data volume.