In one embodiment, one or more portions of video content that were encoded by a video encoder are identified. The one or more portions of video content are to be analyzed for improving encoding performance of the video encoder. The one or more portions of the video content are stored. During an idle time period of the video encoder, the one or more portions of video content are analyzed in combination with a test set stored by the video encoder in order to modify one or more functions of an encoding process used by the video encoder. Based on the analysis, one or more modifications to the encoding process are determined that result in improved performance of the video encoder.