A buffer classification system calculates first flow metrics for a first flow in response to receiving first packet level data. The system in response to receiving HTTP information, generates buffer emulation data and creates one or more data training sets using the first flow metrics and buffer emulation data, wherein training data used to create the one or more training data sets is further generated by computing an average throughput per second observed over different time windows during a video playback. The system in response to creating the one or more training data sets, generates one or more classifier rules used to determine a buffer condition of a flow. The system in response to receiving second packet level data, calculates second flow metrics for a second flow and predicts a buffer condition for the second flow based on the second flow metrics and the one or more classifier rules.