A video residual decoding apparatus is used for applying residual decoding to a transform block that is divided into sub-blocks, and includes a residual decoding circuit and a storage device. The residual decoding circuit enters a coefficient loop for decoding one or more syntax elements at each of coefficient positions within a sub-block that has at least one non-zero coefficient level. The coefficient loop includes one decoding pass and at least one other decoding pass. During the at least one other decoding pass, the residual decoding circuit records side information in the storage device, where the side information is indicative of specific coefficient positions at which specific syntax elements need to be decoded in the one decoding pass. During the one decoding pass, the residual decoding circuit refers to the side information for decoding the specific syntax elements at the specific coefficient positions, respectively.