In a motion vector estimating apparatus, a current picture storage unit stores image data of a current picture, and a reference picture storage unit stores image data of a reference picture. A search window determining unit determines estimation history from previously estimated motion vectors, and determines a search window based on the estimation history. At least one of a shape, size and position of the search window is determined based on the estimation history. The search window is composed of rectangular reference regions. A block matching circuit for performing a block matching process to a current block and each of reference blocks of the search window to determine a motion vector. The search window may be limited in units of pixels, or a load of the apparatus, a power supply voltage or a temperature of the block matching circuit.