Techniques and tools for encoding and decoding a block of frequency coefficients are presented. An encoder selects a scan order from multiple available scan orders and then applies the selected scan order to a two-dimensional matrix of transform coefficients, grouping non-zero values of the frequency coefficients together in a one-dimensional string. The encoder entropy encodes the one-dimensional string of coefficient values according to a multi-level nested set representation. In decoding, a decoder entropy decodes the one-dimensional string of coefficient values from the multi-level nested set representation. The decoder selects the scan order from among multiple available scan orders and then reorders the coefficients back into a two-dimensional matrix using the selected scan order.