Video frames are rotated for encoding and decoding blocks therein. During an encoding process, a number of degrees to rotate a video frame is determined. The video frame is rotated to produce rotated pixel values. A prediction residual is then generated for a block to be encoded based on the rotated pixel values. That block is encoded based on the prediction residual. During a decoding process, an encoded block is decoded to produce inverse transform pixel values, and a number of degrees to inverse rotate an encoded video frame is determined. The encoded video frame is inverse rotated to produce inverse rotated pixel values corresponding to the inverse transform pixel values. The inverse rotated pixel values are then outputted to an output video stream. One or more syntax elements indicating the number of degrees can be encoded to a bitstream during encoding and decoded from the bitstream during decoding.