System and method for improving operational efficiency of a video encoding pipeline used to encode image data. The video encoding pipeline includes a mode decision block, which selects a first inter-frame prediction mode used to prediction encode a first prediction unit, and a motion estimation block, which receives the first inter-frame prediction mode as feedback from the mode decision block when processing a second prediction unit; determines an initial candidate inter-frame prediction mode of the second prediction unit based at least in part on the first inter-frame prediction mode; and determines a final candidate inter-frame prediction mode of the second prediction unit by performing a first motion estimation search based at least in part on the initial candidate inter-frame prediction mode. The mode decision block determines a rate-distortion cost associated with the final candidate inter-frame prediction mode and a prediction mode used to prediction encode the second prediction unit based at least in part on the rate-distortion cost.