Motion information for an internal sub-block of a larger block can be derived for use in encoding or decoding the video block or a coding unit by using the motion information for sub-blocks on the left or top edge of the coding block. The left column of edge sub-blocks and the top row of sub-blocks has motion information, such as motion vectors, derived using such techniques as template matching. The motion vectors of these edge sub-blocks are used in deriving the motion vectors of internal sub-blocks, which leads to better prediction and improved coding efficiency. In another embodiment, other motion information for internal sub-blocks is derived from corresponding information of the edge sub-blocks.