Aspects of the disclosed technology relate to techniques for determining canonical forms of layout patterns. Coordinates of vertices of geometric elements in a window of a layout design are first transformed into new coordinates of the vertices, wherein the coordinates of vertices do not comprise clipped coordinates and the transforming comprises: performing a translation on the coordinates of vertices based on differences between maximum and minimum X/Y coordinate values of the vertices. Based on sums of X/Y coordinate values of the new coordinates of the vertices, a canonical form of the geometric elements is determined. The canonical form coordinates of the vertices may then be determined and sorted. The sorted canonical form coordinates may be employed for pattern matching.