Even when a local area is varied, degradation in recognition accuracy and detection accuracy is suppressed. To that end, a pattern processing apparatus includes a reference local area setting portion 1802 for setting a reference local area based on the detection result of a feature point by a face organ feature point detecting portion 101, a varied local area generating portion 1803 for generating a plurality of varied local area patterns by referring to an image area near the reference local area, a similarity calculating portion 106 for calculating similarities in the reference local areas and in the varied local area patterns between the input pattern and the registered pattern, a representative similarity calculating portion 107 for calculating representative similarity from among the similarities, and a classifying portion 109 for determining a class to which the input pattern belongs.