A video encoded as a bit stream is decoded by maintaining a set of dictionaries generated from decoded prediction residual signals, wherein elements of the set of dictionaries have associated indices. A current macroblock is entropy decoded and inverse quantized to produce decoded coefficients. For the current macroblock, a particular dictionary of the set of dictionaries is selected according to a prediction mode signaled in the bit stream, and particular elements of the particular dictionary are selected according to a copy mode signal in the bit stream and the associated index. The particular elements is scaled and combined, using the decoded coefficients, to reconstruct a current decoded macroblock prediction residual signal. Then, the current decoded macroblock prediction residual signal is combined with previously decoded macroblocks to generate an output macroblock of a reconstructed video, wherein the steps are performed in a decoder.