Data stored in non-volatile storage is decoded using iterative probabilistic decoding. An error correcting code such as a low density parity check code may be used. In one approach, initial reliability metrics, such as logarithmic likelihood ratios, are used in decoding sensed states of a set of non-volatile storage elements. The decoding attempts to converge by adjusting the reliability metrics for bits in code words which represent the sensed state. Simulated annealing using an adjustable temperature parameter based on a level of error in the data read from the system can be performed to assist the iterative decoding process. The simulated annealing can introduce randomness, as noise for example, into the metric based decoding process. Moreover, knowledge of the device characteristics can be used to guide the simulated annealing process rather than introducing absolute randomness. The introduction of a degree of randomness adds flexibility during the iterative decoding that permits possible faster convergence times and convergence in situations where data may otherwise be uncorrectable.