One embodiment of the present invention sets forth a technique for adapting playback bit rate in a content delivery system based on scene complexity of the video content as well as network conditions and other performance factors. A scene complexity map of the video content indicates the complexity levels of different scenes within the video content. Using the scene complexity map, a content player may download scenes of lower scene complexity levels from video streams encoded to lower bit rates to manage the bandwidth consumed to download the video content and allow for the downloading of scenes of higher complexity levels from video streams encoded to higher bit rates.