Embodiments disclosed herein provide systems, methods, and computer readable media that perform adaptive management of a media buffer. In a particular embodiment, a method provides receiving packets representing a media stream into the adaptive media buffer, wherein the media stream comprises a sequence of media segments structured into segment layers, and detecting loss of one or more of the packets representing one of the media segments. The method further provides estimating a round trip delay for packets between the adaptive media buffer and a sender of the media stream and determining a current depth of the media segments within the adaptive media buffer. In response to detecting the packet loss, the method provides discarding media segments within the adaptive media buffer based on the round trip delay, the current depth, and a layer of the segment layers into which each of the one or more media segments is included.