A residual processing circuit has a single-path pipeline and a single-path controller. The single-path pipeline has an inverse scan (IS) circuit, an inverse quantization (IQ) circuit and an inverse transform (IT) circuit arranged to process a current non-zero residual data block in a pipeline manner. The current non-zero residual data block is at least a portion of a transform unit. The single-path controller controls pipelined processing of the current non-zero residual data block at the single-path pipeline. The single-path controller instructs the IS circuit to start IS processing of a next non-zero residual data block before the IT circuit finishes a first half of IT processing of the current non-zero residual data block.