Media content is streamed from a server computer to a client computer. A media file format is used to store data for multiple timeline-altered streams that provides support for switching between the different timeline-altered streams during their presentation. In one aspect, a user can switch between different timeline-altered streams. Upon receiving a user request to switch to a particular timeline-altered stream (the target stream), the client computer accesses a time code stream data object corresponding to the current data unit being presented. The time code stream data object identifies a primary stream presentation time corresponding to the data unit. The client then uses the primary stream presentation time to index into a table of mappings to byte offsets. The table provides a mapping of the primary stream presentation time to a corresponding byte offset of the target stream.