A method for dynamically adjusting jitter buffer size according to buffer fill dynamics is disclosed. In one embodiment, an upper threshold and lower threshold for the jitter buffer are identified, wherein the lower buffer threshold identifies a minimum desirable number of packets in the jitter buffer, and the upper buffer threshold identifies a maximum desirable number of packets in the jitter buffer. Operating characteristics of the jitter buffer are monitored to identify instances when the jitter buffer size falls below or exceeds the desired thresholds. When a threshold is crossed, the adaptive algorithm alters the playback offset time, by introducing or deleting packets into the transmission path, to allow the jitter buffer size to return to a desirable target size within the threshold boundaries.