An apparatus and method for recognizing a specific area of an image in a portable terminal. More particularly, an apparatus and method are for determining feature points with very high similarities as one group when the portable terminal recognizes a building included in an image or a picture, and for estimating a matching relation of the group to improve building recognition performance. The apparatus includes an image analyzer configured to, upon extracting feature points used for building recognition, classify feature points with similarities among the extracted feature points into a group, and recognize a building after estimating a matching relation by regarding the classified group as a feature point.