A partial decoding circuit with inverse second transform has a transpose buffer, a first-direction inverse residual transform circuit, and a second-direction inverse residual transform circuit. The transpose buffer stores an intermediate inverse residual transform result. The first-direction inverse residual transform circuit processes an inverse quantization output to generate the intermediate inverse residual transform result to the transpose buffer. The second-direction inverse residual transform circuit accesses the transpose buffer to retrieve the intermediate inverse residual transform result, and processes the intermediate inverse residual transform result to generate a final inverse residual transform result, where the final inverse residual transform result of the inverse second transform is further processed by an inverse transform circuit. The first-direction inverse residual transform circuit and the second-direction inverse residual transform circuit process partial residual transform data of different process units in a parallel processing manner.