A method in a scalable encoder for reducing the average latency associated with randomly accessing an encoded digital video signal is disclosed. This method may include converting a digital video signal into a first layer having a first degree of quality and a second layer having a second degree of quality that is higher than the first degree of quality. The method may also include encoding the first layer at a first intra-frame rate, and encoding the second layer at a second intra-frame rate that is lower than the first intra-frame rate. A method in a scalable decoder for reducing the average latency associated with randomly accessing an encoded digital video signal is also provided. The method includes accessing a first layer of a digital video signal. The first layer includes the digital video signal encoded at a first degree of quality and a first intra-frame rate. The method also includes accessing a second layer of the digital video signal. The second layer includes the digital video signal encoded at a second degree of quality higher than the first degree of quality and a second intra-frame rate lower than the first intra-frame rate.