An embodiment of the invention provides a method performed by a de-multiplexer that is coupled to a front-end stream buffer and a plurality of back-end video buffers. The front-end stream buffer is configured to buffer a plurality of video data units of a video stream. The video stream includes a plurality of subordinate video streams interleaved therein. Each of the back-end video buffers is configured to buffer the video information contained in a corresponding one of the subordinate video streams. First, the de-multiplexer determines whether it is blocked from fetching the video information of a first video data unit buffered in the front-end stream buffer. Then, if being blocked, the de-multiplexer skips accessing the first video data unit and instead fetches the video information of a second video data unit buffered in the front-end stream buffer.