A method to map a plurality of feature maps of a neural network onto a memory hierarchy includes mapping a first feature map of the plurality of feature maps to a memory in a memory hierarchy having available memory space and providing quickest access to the first feature map. The method also includes, when the first feature map expires, removing the first feature map from the memory used to store the first feature map.