This disclosure describes video encoding and decoding techniques in which a first order prediction process and a second order prediction process are used in combination to generate predictive video blocks for video coding. First order prediction may be similar to conventional motion estimation and motion compensation that generates residual video blocks. The second order prediction may involve a process similar to conventional intra-prediction, but is performed on the residual video blocks. The techniques of this disclosure may pre-define the second order prediction to a specific mode, such as a mode similar to the intra-DC mode used in intra coding. In addition, the techniques of this disclosure may combine aspects of the first order and second order prediction into a single process so that the effects of second order prediction on the residuals are taken into account during the first order prediction process, which may improve compression.