An encoder for encoding a video by performing motion estimation using predicted residuals in the rate-distortion cost expression. The predicted residuals are determined by finding a predicted motion vector with respect to the original reference frame and calculating the corresponding residual values. The actual motion vector is then selected based on minimization of the rate-distortion cost expression with respect to the reconstructed reference frame, where the rate-distortion cost expression includes the predicted residuals. In particular, the cost expression includes reconstructed predicted residuals within the distortion term.