A video coder, such as a video encoder or a video decoder, may use reference pixels in a first picture to pad pixels outside a picture boundary of a second picture. The second picture is a different picture from the first picture. The padded pixels are in a padding area surrounding the second picture. The video coder may encode or decode one or more blocks of the video data based on the padded pixels.