Described is a method and system for embedding unsupervised learning into three critical processing stages of the spatio-temporal visual stream. The system first receives input video comprising input video pixels representing at least one action and at least one object having a location. Microactions are generated from the input image using a set of motion sensitive filters. A relationship between the input video pixels and the microactions is then learned, and a set of spatio-temporal concepts is learned from the microactions. The system then learns to acquire new knowledge from the spatio-temporal concepts using mental imagery processes. Finally, a visual output is presented to a user based on the learned set of spatio-temporal concepts and the new knowledge to aid the user in visually comprehending the at least one action in the input video.