An apparatus for visual search and retrieval using sematic information is described herein. The apparatus includes a controller, a scoring mechanism, an extractor, and a comparator. The controller is to segment an incoming video stream into a plurality of activity segments, wherein each frame is associated with an activity. The scoring mechanism is to calculate a score for each segment, wherein the score is based, at least partially, on a classification probability of each frame. The extractor is to extract deep features from a highest ranked segment, and the comparator is to determine the top-K neighbors based on the deep features.