An example device for decoding video data includes a processing system comprising one or more processors implemented in circuitry and configured to: determine that a first temporal layer identifier of a first picture of the video data is included in a first set of temporal layers; in response to the first temporal layer identifier being included in the first set of temporal layers, decode blocks of the first picture on a block by block basis; determine that a second temporal layer identifier of a second picture of the video data is included in a second set of temporal layers, the second set of temporal layers being higher than the first set of temporal layers; and in response to the second temporal layer identifier being included in the second set of temporal layers, execute a neural network-based video decoder to decode the second picture.