A technique for encoding and decoding video information uses a plurality of video processing modules (VPMs), whereby each video processing module is dedicated to a particular video processing function, such as filtering, matrix arithmetic operations, and the like. Information is transferred between the video processing modules using a set of first-in first-out (FIFO) buffers. For example, to transfer pixel information from a first VPM to a second VPM, the first VPM stores the pixel information at the head of a FIFO buffer, while the second VPM retrieves information from the tail of the FIFO buffer. The FIFO buffer thus permits transfer of information between the VPMs without storage of the information to a cache or other techniques that can reduce video processing speed.