Systems and methods to process motion vectors of video data are disclosed. According to an embodiment, an intra-block in a video frame of the video data is identified. At least a first set and a second set of inter-blocks that neighbor the identified intra-block are identified, where each inter-block in a set of inter-blocks has a motion vector associated therewith. Data of the first and second set of inter-blocks is evaluated to determine an error associated with each of the first and second sets. A motion vector associated with motion vectors of the inter-blocks of the set with the least error is determined. The determined motion vector may be associated with the identified intra-block.