A device, an encoding method, and a decoding method enable a separate marking of base representations and enhanced representations of key access units to save memory or to allow a better scalable video coding optimization. The encoding method of a sequence of original pictures to a sequence of access units includes, after encoding of one of the access units, storing a first decoded picture of the first encoded picture and a second decoded picture of the second encoded picture for inter prediction for encoding of others of the access units; and identifying the first decoded picture and the second decoded picture to be no longer used for inter prediction. The decoding method includes decoding the first access unit, where a first decoded picture is decoded from the first picture and a second decoded picture is decoded from the second picture; marking the first and second decoded pictures as used for inter prediction; decoding the second access unit; and marking one of the first and second decoded pictures as no longer used for inter prediction.