Techniques for advanced residual prediction (ARP) in video coding may include receiving a first encoded block of video data in a first access unit, wherein the first encoded block of video data was encoded using advanced residual prediction and bi-directional prediction, determining temporal motion information for a first prediction direction of the first encoded block of video data, and identifying reference blocks for a second prediction direction, different than the first prediction direction, using the temporal motion information determined for the first prediction direction, wherein the reference blocks are in a second access unit.