A method of decoding video data includes decoding a first block of video data to produce a block of reconstructed luma residual values and a block of predicted chroma residual values, wherein the block of video data has one of a 4:2:0 or a 4:2:2 chroma sub-sampling format. The method further includes performing a color residual prediction process to reconstruct a block of chroma residual values for the first block of video data using a subset of the reconstructed luma residual values as luma predictors for the block of predicted chroma residual values.