Buffer optimization techniques are described herein in which a graphics processing system is configured to implement and select between a plurality of buffer schemes for processing of an encoded data stream in dependence upon formats used for decoding and rendering (e.g., video format, bit depth, resolution, content type, etc.) and device capabilities such as available memory and/or processing power. Processing of an encoded data stream for display and rendering via the graphics processing system then occurs using a selected one of the buffer schemes to define buffers employed for the decoding and rendering, including at least configuring the sizes of buffers. The plurality of schemes may include at least one buffer scheme for processing the encoded content when the input format and the output format are the same, and a different buffer scheme for processing the encoded content when the input format and the output format are different.