The application of a watershed algorithm to pixels and their touch values obtained from a scan of a touch sensor panel to determine patches corresponding to images of touch is disclosed. Prior to applying the watershed algorithm, background pixels having little or no touch values can be eliminated. A primary merge algorithm can then merge adjacent patches together when the saddle point between them is shallow as compared to the peak represented by the patches. However, if two candidate patches for merging have a total number of pixels below a certain threshold, these two patches may not be merged under the assumption that the patches might have been caused by different fingertips. Conversely, if two candidate patches for merging have a total number of pixels above a certain threshold, these two patches can be merged under the assumption that the patches were caused by a single thumb or palm.