In order to utilize memo data written by hand for purposes of retrieving a document, the original document image data is stored and, thereafter, a document image data with memo is separately input, the document image data with memo being comprised of the original document image onto which is additionally written memo data by hand. The position of the document image data with memo is then brought into alignment with the original document image data and is correlated to extract only that memo data which has been written by hand. Then the kind of memo is discriminated, the process is effected depending upon the kind of memo, and the memo data is stored in a secondary data file for retrieval. To retrieve the document, the memo data of a plurality of documents is read out from the secondary data file depending upon the kind of memo that is designated, and is displayed as a look-up table. If the operator designates any one of them, the image of the corresponding original document is read out from the original document image file and is displayed.