In an example method, a system receives a plurality of frames of a video, and generates a data structure representing the video and representing a plurality of temporal layers. Generating the data structure includes: (i) determining a plurality of quality levels for presenting the video, where each of the quality levels corresponds to a different respective sampling period for sampling the frames of the video, (ii) assigning, based on the sampling periods, each of the frames to a respective one of the temporal layers of the data structure, and (iii) indicating, in the data structure, one or more relationships between (a) at least one the frames assigned to at least one of the temporal layers of the data structure, and (b) at least another one of the frames assigned to at least another one of the temporal layers of the data structure. Further, the system outputs the data structure.