The image signature extraction device includes an image signature generation unit and an encoding unit. The image signature generation unit extracts region features from respective sub-regions in an image in accordance with a plurality of pairs of sub-regions in the image, the pairs of sub-regions including at least one pair of sub-regions in which both a combination of shapes of two sub-regions of the pair and a relative position between the two sub-regions of the pair differ from those of at least one of other pairs of sub-regions, and based on the extracted region features of the respective sub-regions, generates an image signature to be used for identifying the image. The encoding unit encodes the image signature.