Some embodiments provide a method of aligning a pair of images. The method defines multiple different pairs of images at multiple different resolutions. The method hierarchically aligns the original pair of images by first aligning the pair of images at the lowest resolution and then aligning each pair of images at each higher resolution based on the alignments of the pair of images at the lower resolutions. For some of the resolutions, to perform the hierarchically alignment, the method identifies, for at least one image at each resolution, portions that are suitable for performing the alignment and portions that are not suitable for performing the alignment. The method compares each pair of images at a particular resolution by using the suitable portions while excluding the unsuitable portions from the comparison.