The present invention relates to a data processing device (10) comprising a processing unit (12) and a memory unit (14), and to a method for controlling operation of a memory unit (14) of a data processing device. The memory unit (14) comprises a main memory (16), a low- level cache memory (20.2), which is directly connected to the processing unit (12) and adapted to hold all pixels of a currently active sliding search area for reading access by the processing unit (12), a high-level cache memory (18), which is connected between the low-level cache memory and the frame memory, and a first pre-fetch buffer (20.1), which is connected between the high-level cache memory and the low- level cache memory and which is adapted to hold one search-area column or one search-area line of pixel blocks, depending on the scan direction and scan order followed by the processing unit. Reading and fetching functionalities are decoupled in the memory unit (14). The fetching functionality is concentrated on the higher cache level, while the reading functionality is concentrated on the lower cache level. This way concurrent reading and fetching can be achieved, thus enhancing the performance of a data processing device.