A method and apparatus are disclosed for performing motion estimation and compensation to fractional pixel accuracy using polyphase prediction filters as part of a video compression/decompression technique. A motion estimator applies a set of polyphase filters to some data in the reference picture and generates motion vectors, an estimated macroblock of video data, and a residual error macroblock of video data. The data referenced in the reference picture usually have more data than a macroblock since multi-tap filtering needs to access more data. A motion compensator generates a compensated macroblock of video data in response to the reference video data, the residual error macroblock of video data, and a polyphase prediction filter decided by the motion vector. The reference video data are usually reconstructed at the compensator side.