Systems and methods for performing video understanding and analysis. Sets of feature maps for high resolution images and low resolution images in a time sequence of images are combined into combined sets of feature maps each having N feature maps. A time sequence of temporally aggregated sets of feature maps is created for each combined set of feature maps by: selecting a selected combined set of feature maps corresponding to an image at time “t” in the time sequence of images; applying, by channel-wise multiplication, a feature map weighting vector to a number of combined sets of feature maps that are temporally adjacent to the selected combined set of feature maps; and summing elements of the number of combined set of feature maps into a temporally aggregated set of feature maps. The time sequence of temporally aggregated sets of feature maps is processed to perform video understanding processing.