This disclosure describes techniques to improve a user experience in a Wireless Display (WD) system. The WD system includes a source device that provides media data to one or more sink devices. The techniques are directed toward reducing end-to-end latency in the WD system while improving video playback quality at the sink devices. More specifically, the techniques include customized buffering at the sink devices based on application awareness for the media data. The techniques include learning the type of application for the media data, and adjusting the size of buffers in the processing pipeline to achieve an appropriate balance between smoothness and latency for the application type. For example, when the media data is for a video playback application, the techniques include increasing the buffer size to increase smoothness in the video playback application.