A microprocessor employs an L0 cache. The L0 cache is located physically near the execute units of the microprocessor and is relatively small in size as compared to a larger L1 data cache included within the microprocessor. The L0 cache is accessed for those memory operations for which an address is being conveyed to a Load/store unit within the microprocessor during the clock cycle in which the memory operation is selected for access to the L1 data cache. The address corresponding to the memory operation is received by the L0 cache directly from the execute unit forming the address. If a hit in the L0 cache is detected, the L0 cache either forwards data or stores data corresponding to the memory operation (depending upon the type of the memory operation). The memory operation is conveyed to the L1 data cache in parallel with the memory operation accessing the L0 cache. If the memory operation misses in the L0 cache and hits in the L1 data cache, the cache line corresponding to the memory operation may be conveyed to the L0 cache as a line fill.