Methods, systems, and processor-readable media for pruning a training dictionary for use in detecting anomalous events from surveillance video. Training samples can be received, which correspond to normal events. A dictionary can then be constructed, which includes two or more classes of normal events from the training samples. Sparse codes are then generated for selected training samples with respect to the dictionary derived from the two or more classes of normal events. The size of the dictionary can then be reduced by removing redundant dictionary columns from the dictionary via analysis of the sparse codes. The dictionary is then optimized to yield a low reconstruction error and a high-interclass discriminability.