A cache memory system uses multi-bit Error Correcting Code (ECC) with a low storage and complexity overhead. In an embodiment, error correction logic may include a first error correction logic to determine a number of errors in data that is stored in a cache line of a cache memory, and a second error correction logic to receive the data from the first error correction logic if the number of errors is determined to be greater than one and to perform error correction responsive to receipt of the data. The cache memory system can be operated at very low idle power, without dramatically increasing transition latency to and from an idle power state due to loss of state. Other embodiments are described and claimed.