A system and method for signaling low-to-high layer switching points in a file format level to enable efficient scalable stream switching in streaming servers and local file playback. The present invention also provides for a system and method for signaling low-to-high layer switching points in video bit stream, e.g., to enable intelligent forwarding of scalability layers in media-aware network elements or computationally scalable decoding in stream recipients.