A video decoder may be configured to receive a block of video data that was encoded using a coding mode that includes a search process in one or more reference frames. The video decoder may prefetch reference samples in a fixed search region of at least one reference frame of the one or more reference frames, and decode the block of video data using the coding mode, including performing the search process for the coding mode using the prefetched reference samples.