In a dictionary image generating section, images of objects each belonging to any one of L categories are taken by a camera from predetermined M directions, a matching region of the object to be recognized is extracted from each of the object images, and dictionary images (N=L×M, n=1, 2, . . . , N) which are the representatives of combinations of direction and category are generated and stored together with the directions of the objects. When a test image is given, the degree of similarity r(n, X, Y) between an n-th dictionary image and the image region at the matching position (X, Y) in the test image is computed by a matching section. This matching process is repeated with the matching position (X, Y) and the number n of the dictionary image being varied, and the matching position (Xmax, Ymax) where the degree of similarity r(n) becomes the highest is detected.