A video processing apparatus with residue prediction includes a motion estimation/compensation unit to determine a matching block of a reference video frame, obtain a motion vector of a current block of a current video frame that is related to the matching block, and acquire neighboring reconstructed pixels adjacent to the current block and corresponding pixels adjacent to the matching block with the motion vector alignment. Additionally, a pseudo-residue generating unit is included and constructs pseudo residues according to the neighboring reconstructed pixels and the corresponding pixels, an arithmetic unit is included and generates first-order residues by subtracting the matching block from the current block, and a residue-predicting unit is included and derives second-order residues and corresponding information according to the pseudo residues and the first-order residues. Moreover, a post-processing unit is included and derives a reconstructed current block according to the second-order residues and its corresponding information.