Techniques are presented herein for switching between a software decoder and a hardware decoder in a hybrid decoder. A video bit stream is received. The video bit stream is analyzed to determine whether or not the video bit stream can be decoded by the hardware decoder. When the video bit stream can be decoded by the hardware decoder, the video bit stream is routed to the hardware decoder and also stored in a first buffer. The video bit stream is decoded by the hardware decoder and when a reference frame is decoded, the reference frame is stored in a second buffer. When the video bit stream cannot be decoded by the hardware decoder, the video bit stream is routed to the software decoder for decoding. When an instantaneous decoder refresh (IDR) frame is decoded by the software decoder, the IDR frame is stored in the first buffer.