Techniques and tools for representing pixel data in a video processing or capture system are described. Described techniques and tools provide efficient color representation for video processing and capture, and provide flexibility for representing colors using different bit precisions and memory layouts. Described techniques and tools include video formats that can be used, for example, in hardware or software for capture, processing, and display purposes. In one aspect, chroma and luma information for a pixel in a video image is represented in a 16-bit fixed-point block of data having an integer and fractional components. Data can be easily converted from one representation to another (e.g., between 16-bit and 10-bit representations). In other aspects, formats for representing 8-, 10- and 16-bit video image data (e.g., packed and hybrid planar formats), and codes for indicating the formats, are described.