A residual up-sampling apparatus has a residual up-sampling buffer and a shared residual up-sampling circuit. The residual up-sampling buffer stores an intermediate residual up-sampling result. The shared residual up-sampling circuit employs a same processing kernel to perform a first-direction residual up-sampling operation and a second-direction residual up-sampling operation. The first-direction residual up-sampling operation processes an inverse transform output of an inverse transform circuit to generate the intermediate residual up-sampling result to the residual up-sampling buffer. The second-direction residual up-sampling operation performs transpose access upon the residual up-sampling buffer to retrieve the intermediate residual up-sampling result, and processes the intermediate residual up-sampling result to generate a final residual up-sampling result.