A method and apparatus of adaptive interpolation filter for motion compensation with fractional-pixel accuracy are disclosed. Embodiments of the present invention generate interpolated reference samples at non-integer locations based on existing reference samples in a reference block by using an interpolation filter set adaptively according to pixel location and/or pixel characteristics. A current block is then encoded or decoded using a temporal predictor including the interpolated reference samples. The adaptive interpolation filter can be applied to the prediction unit (PU). In one embodiment, the interpolation filter set consists of interpolation filters having different filter lengths. An interpolation filter with a longer tap length is applied to generate interpolated reference samples farther from the block boundary and an interpolation filter with a shorter tap length is applied to generate interpolated reference samples closer to the block boundary.