To address limitations of conventional packet buffer management strategies, the various embodiments provide methods implemented by a processor operating on a VoIP receiver device for managing media packet rejection logic on the receiving device to improve audio playout. The embodiment methods enhance the likelihood that a receiver device will buffer (rather than reject) an incoming media packet by maintaining a history of previously rejected media packets (i.e., a packet rejection history) and referencing the packet rejection history when determining whether to reject or to accept an incoming packet the receiver device has just received. Thus, the various embodiments may reduce gaps in playback by avoiding unnecessary packet rejections and may promote an overall increase in playout quality.