Memory eviction that recognizes not all evictions have an equal cost on system performance. A management device keeps a weight and/or a count associated with each portion of memory. Each memory portion is associated with a source agent that generates requests to the memory portion. The management device adjusts the weight by a cost factor indicating a latency impact that could occur if the evicted memory portion is again requested after being evicted. The latency impact is a latency impact for the associated source agent to replace the memory portion. In response to detecting an eviction trigger for the memory device, the management device can identify a memory portion having a most extreme weight, such as a highest or lowest value weight. The management device replaces the identified memory portion with a memory portion that triggered the eviction.