A scene matching reference data generation system inputs a set of probe data. The set of the probe data includes captured images sequentially obtained by a plurality of probe cars and the vehicle positions of the probe cars. The system temporarily stores the captured images, evaluates image similarity degrees of the captured images, and assigns the similarity degrees to the captured images. The system selects, as a plurality of processing target captured images, a plurality of the captured images having similarity degrees equal to or higher than a first predetermined degree, determines a representative image-capturing position that is a representative of the image-capturing positions of the plurality of the processing target captured images, generates image feature point data based on the plurality of the processing target captured images, and generates the reference data for scene matching by associating the image feature point data with the representative image-capturing position.