A method and system are provided for image matching using an appearance augmented 3-D point cloud that includes receiving a first and second posed image, extracting image features from the first and second posed image, and comparing the extracted image features from the first posed image with the extracted image features from the second posed image. Next, a comparison is done to identify one or more matched features of the first and second posed images based on a feature appearance, where a grouping of the matched features of the first and second posed images is performed where the grouped matched features are associated with a first three-dimensional point. A position of the first three-dimensional point is identified based on a positional triangulation of the grouped matched features associated with the first three-dimensional point, and the first three-dimensional point is then augmented with the grouped matched features.