Disclosed is a prefetch execution unit, prefetch instruction buffer and a prefetch victim buffer which operate to optimize prefetching by recording a particular cache miss' history. To record cache misses, victimized (overwritten) lines and/or a prefetch tag are stored in a prefetch victim buffer. When the processor experiences a cache miss, it accesses the prefetch victim buffer to retrieve information relating to the prefetch victim. The prefetch execution unit then modifies the values of the additional field or fields and then stores the modified augmented prefetch instruction in the prefetch instruction buffer. The next time a prefetch instruction for the victimized lines is executed by the processor, the new values of the modified augmented prefetch instruction will dictate where the prefetch information is stored or what size increment it has. By continuous modification of the augmented prefetch instructions, eventually thrashing may be eliminated.