In one example, a video coder, such as a video encoder or video decoder, is configured to code a video parameter set (VPS) for one or more layers of video data, wherein each of the one or more layers of video data refer to the VPS, and code the one or more layers of video data based at least in part on the VPS. The video coder may code the VPS for video data conforming to High-Efficiency Video Coding, Multiview Video Coding, Scalable Video Coding, or other video coding standards or extensions of video coding standards. The VPS may include data specifying parameters for corresponding sequences of video data within various different layers (e.g., views, quality layers, or the like). The parameters of the VPS may provide indications of how the corresponding video data is coded.