A secondary curve, the ends of which coincide with the inner corner and the outer corner of the eye, is determined successively, and the total of the edge values of the pixels overlapping the secondary curve is calculated as an evaluation value. Next, a characteristic curve is generated on the basis of data made up of the calculated evaluation value and the Y-coordinate of the intersection between the secondary curve and a straight line passing through the center of a line segment whose ends coincide with the inner corner and the outer corner of the eye. Then, the reference positions for the upper eyelid and the lower eyelid of the eye are set on the basis of the result of an attempt to detect a pixel group occurring because of the red-eye effect in a search area defined on the basis of peaks in the characteristic curve.