To improve feature selection accuracy during a visual search, interest points within a query image are two-way matched to features in an affine transformed image or otherwise transformed version of the query image. A user device implements a method for selecting local descriptors in the visual search. The method includes: detecting a first set of interest points for the original image; computing an affine transform matrix; computing a new image as a transformation of the original image using the affine transform matrix; detecting a second set of interest points from the and new image; performing a two-way matching between the first set of interest points and the second set of interest points; sorting matching pairs according to a specified self-matching score (SMS); assigning an infinite value to SMS of unmatched interest points from the original image; selecting the interest points based on SMS. Significant performance gains reduce false positive matches.