Techniques and tools are described for decoding video data having samples that have been scaled in the spatial domain. For example, a decoder receives a bit stream that includes coded video data for a current frame. The decoder processes at least one syntax element (e.g., sequence layer flag, frame layer flag) that indicates whether the current frame should be scaled up in value in a spatial domain. If so, then the samples for the current frame are scaled up in value in the spatial domain. As another example, for a reference frame used in motion compensation for a current frame, a decoder scales samples of the reference frame so the range of the reference frame matches the range of the current frame.