A video residual decoding apparatus includes a residual decoding circuit and a neighbor storage device. The residual decoding circuit applies residual decoding to a transform block, wherein the transform block is divided into M sub-blocks, M is a positive integer, and the M sub-blocks are processed by the residual decoding in a diagonal scan order. The neighbor storage device stores neighbor data that belong to neighboring sub-blocks and are referenced by the residual decoding of a current sub-block, wherein neighbor data belonging to a sub-block is derived from a residual decoding result of the sub-block, and a storage size of the neighbor storage device is not larger than a maximum data amount of neighbor data derived from residual decoding results of N sub-blocks, where N is a positive integer, and N is smaller than M.