An example device for decoding video data includes: a memory configured to store video data; and a processing system comprising one or more processors implemented in circuitry, the processing system being configured to: determine whether motion information of a block of video data is for sub-blocks of the block larger than individual pixels of the block or for the individual pixels, the block being associated with data indicating that the block is to be predicted using affine motion compensation; in response to determining that the motion information of the block is for the sub-blocks, perform sub-block-based affine motion compensation to form a prediction block for the block; in response to determining that the motion information is for the individual pixels, perform pixel-based affine motion compensation to form the prediction block for the block; and decode the block using the prediction block.