An encoder configured to receive an input video comprising respective frames, each frame being divided into a plurality of tiles and each tile being divided into a plurality of blocks. The encoder is configured to generate a base encoded stream using a base encoder, determine a temporal mode for one or more further encoded enhancement streams generated using an enhancement encoder and generate the one or more further encoded enhancement streams according to the determined temporal mode. The temporal mode is either a first temporal mode that does not apply non-zero values from a temporal buffer or a second temporal mode that does apply non-zero values from the temporal buffer. Generating the one or more further encoded enhancement streams comprises applying a transform to each of a series of blocks. The temporal mode is determined for one or more of a frame, tile or block.