Described is a system for real-time object recognition. During operation, the system extracts convolutional neural network (CNN) feature vectors from an input image. The input image reflects a scene proximate the system, with the feature vector representing an object in the input image. The CNN feature vector is matched against feature vectors stored in a feature dictionary to identify k nearest neighbors for each object class stored in the feature dictionary. The matching results in a probability distribution over object classes stored in the feature dictionary. The probability distribution provides a confidence score that each of the object classes in the feature dictionary are representative of the object in the input image. Based on the confidence scores, the object in the input image is then recognized as being a particular object class when the confidence score for the particular object class exceeds a threshold.