A motion compensator includes a divider, a frame memory transfer controller, and a motion compensation processor. Based on information about a coding unit CU and prediction unit PU provided by a decoder, the divider determines whether or not to divide the PU. Next, based on a motion vector of the PU yet to be divided, reference image information, and information about divided blocks locations, the frame memory transfer controller determines the storage location of the reference image of a reference picture in a frame memory on the basis of each of the blocks divided, thereby obtaining reference image data. The motion compensation processor performs motion compensation operation on a motion compensation control block basis to generate a predicted image. Then, a reconstructor obtains a restored image based on a residual image generated by an inverse frequency transformer.