A data processor has a main memory (203) which stores data and instructions to be used by the processor, an instruction processor (201) and two cache memories (100,101). The first cache memory (101) is a large capacity port direct mapped cache memory, and the second cache memory (100) is a small capacity two port set associative cache memory. The instruction processor (201) controls the transfer of data to/from the cache memories (100,102) on the basis of instruction from the main memory, so that data needed frequently is stored in the first cache memory (101) and data needed less frequently is stored in the second cache memory (100). With such an arrangement, data stored in the second cache memory (100) can be removed therefrom after it has been accessed, and other data stored therein, thereby increasing the probability that data needed at any time will be in the first or second cache memories (101,100), without storing useless data on the first cache memory (101).