In general, techniques are described for performing residual prediction in video coding. As one example, a device configured to code scalable or multi-view video data may comprise one or more processors configured to perform the techniques. The processors may determine a difference picture, for a current picture, based on a first reference picture in a same layer or view as the current picture and a decoded picture in a different layer or view as the current picture. The decoded picture may be in a same access unit as the first reference picture. The processors may perform bi-prediction based on the difference picture to code at least a portion of the current picture.