A video coder may be configured to code a chroma block of video data as a function of reconstructed luma samples. The video coder may store, during the prediction process for a luma block, a sum of the reconstructed luma sample values for each sub-block of the luma block in a first buffer, and store, during the prediction process for the luma block, an average of the reconstructed luma sample values in a second buffer. The video coder may perform chroma from luma prediction to reconstruct a chroma block of the video data corresponding to the luma block using the sums and the average.