A system and method for determining the image and ancillary image sides of scanned hardcopy media. Hardcopy prints typically provide ancillary data to supply additional information for an area of interest and can enhance the analysis of key data in the print. Postcards, for example, provide a unique form of ancillary data. On the back of a postcard image, ancillary data information such as stamps, user notes, dates, addresses, and other types of annotation can be found. This data typically provides additional information to a reader of the postcard. Dual sided scanners provide both the image and ancillary (i.e., non-image) scans simultaneously. In addition, ancillary data can be in a variety of formats, such as audio, video and the like. When a hardcopy print is scanned the scanner provides two image files that represent the image and ancillary image sides of the print. However, given two image files that represent a hardcopy print scan, the front and back need to be automatically identified as to which is of primary interest to a user.