A method and system for extracting a visual descriptor using a feature selection are provided. The system includes an image input unit configured to receive an image, a candidate feature point group detecting unit configured to detect a point having a local maximum or minimum of local region filtering in scale-space images as being included in a candidate feature point group, a feature point selecting unit configured to calculate an importance for each candidate feature point, depending on its characteristics, select the candidate feature point as a feature point when its importance is greater than the predetermined threshold value, a dominant orientation calculating unit configured to calculate a dominant orientation of the selected feature point and a visual descriptor extracting unit configured to extract a patch for each feature point, according to its scale, location and dominant orientation, and extract a visual descriptor from the patch.