An MPEG video decoder 1 decodes a MPEG video stream using a discrete cosine transform together with a motion compensated prediction performing backward prediction and forward prediction. A frame buffer 104a is provided with a storage area for forward-reference luminance data used for the backward prediction and a storage area for rearward-reference color-difference data used for the forward prediction. A frame buffer 104b is provided with a storage area for forward-reference color-difference data used for the backward prediction and a storage area for rearward-reference luminance data used for the forward prediction. Memory access for each of the frame buffers 104a, 104b with an input/output data bus width of 16 bit is performed in a parallel processing.