A video decoder organizes and stores pixel lines of a reference picture into first and second memory devices. The video decoder then reads portions of a pixel block from the first and second memory devices and processes such a pixel block for generating a subsequent picture. By reading from the first and second memory device with time overlap, latency is minimized for faster video decoding.