Some embodiments described herein cover a machine learning architecture with a separated perception subsystem and application subsystem. These subsystems can be co-trained to yield a lifelong learning system that can capture spatio-temporal regularities in its inputs. In one example embodiment a first node of the machine learning architecture receives a data item. The first node receives a first feature vector that was generated at a first time. The first node processes information from at least a portion of the data item and at least a portion of the first feature vector at a second time to generate a second feature vector. The first node generates a sparse feature vector from the second feature vector, wherein a majority of feature elements in the sparse feature vector have a value of zero. A second node of the machine learning architecture then processes the sparse feature vector to determine a first output.