A video encoder is configured to determine a picture size for one or more pictures included in a video sequence. The picture size associated with the video sequence may be a multiple of an aligned coding unit size for the video sequence. In one example, the aligned coding unit size for the video sequence may comprise a minimum coding unit size where the minimum coding unit size is selected from a plurality of smallest coding unit sizes corresponding to different pictures in the video sequence. A video decoder is configured to obtain syntax elements to determine the picture size and the aligned coding unit size for the video sequence. The video decoder decodes the pictures included in the video sequence with the picture size, and stores the decoded pictures in a decoded picture buffer.