To determine whether mail address characters are printed or handwritten by a single recognizer (OCR, 40) in a mail processing machine, the front surface of a mail is determined by the presence of a postage stamp, an address window, an address label or by comparing the character areas of both surfaces of the mail. Further, when no stamp and no address window/label are attached, the mail address character area is detected by compressing and binarizing the entire front surface image signals. Thereafter, only the address character image signals corresponding to the detected address window/label or mail address character area are processed through the recognizer.