An image capturing and processing apparatus has image pickup means, first image memorizing means in which an object image from a predetermined viewing point position captured by the image pickup means is memorized, three-dimensional shape model memorizing means for producing a reference object image from a viewing point position nearest to the pick-up object image on the basis of a reference three-dimensional shape model, second image memorizing means in which the produced reference object image is memorized, different portion extracting means for extracting the difference between the object image and the reference object image memorized in the respective image memorizing means, and shape model correcting means for correcting the reference three-dimensional shape model on the basis of the extracted difference. By the reference three-dimensional shape model which is the typical shape model of an object being corrected on the basis of the different portion between the object image and the reference object image, the shape model of the object is recovered.