A data processing system includes a cache memory 58 and cache control circuitry 56 for applying a cache replacement policy based upon a retention priority value PV stored with each cache line 66 within the cache memory 58. The initial retention priority value set upon inserting a cache line 66 into the cache memory 58 is dependent upon either or both of which of a plurality of sources issued the access memory request that resulted in the insertion or the privilege level of the memory access request resulting in the insertion. The initial retention priority level of cache lines resulting from instruction fetches may be set differently from cache lines resulting from data accesses.