Disclosed herein is a system and method of identifying misplaced products on a retail shelf using a feature extractor trained to extract features from images of products on the shelf and output identifying information regarding the product in the product image. The extracted features are compared to extracted features in a product library and a best fit is obtained. A misplaced product is identified if the identifying information produced by the feature extractor fails to match the identifying information associated with the best fit features from the product library.