Video data streams can be encoded and decoded using inter or intra prediction. The blocks of a frame can be processed based on depth, from the lowest level sub-blocks to the highest level large blocks, and divided into groups of blocks to be inter predicted, blocks having sub-blocks that are to be inter predicted and sub-blocks that are to be intra predicted, and blocks to be intra predicted, and the blocks to be inter predicted are encoded first, the blocks having sub-blocks to be inter predicted and intra predicted encoded second, and the blocks to be intra predicted encoded last. The availability of data from the inter predicted blocks can improve the performance of intra prediction over processing the blocks in the scan order since more pixel data is available for intra prediction of some blocks.