An information processing device includes: a feature quantity obtaining unit which obtains a feature quantity of an object to be extracted, which is extracted from a retrieval target, and specific information to be specified an appearing location of it; a feature quantity holding unit which, when storing the feature quantity in a feature quantity table, adds new identification information to the feature quantity and holds the feature quantity in the feature quantity table when a similar feature quantity in which a similarity with the feature quantity is no less than a threshold is not included in the feature quantity table, and outputs identification information of the similar feature quantity as identification information of the feature quantity when the similar feature quantity is included in the feature quantity table; and a retrieval table holding unit which holds the specific information associated with the added identification information or the outputted identification information.