In an example, a method of decoding video data includes determining a prediction partitioning structure for predicting pixel values associated with a block of video data. The method also includes determining a transform partitioning structure for applying one or more transforms to the predicted pixel values. Determining the transform split structure includes splitting a parent transform unit, upon determining the transform partitioning structure comprises splitting the parent transform unit into one or more square transforms, determining one or more square transforms such that each of the one or more square transforms correspond to exactly one prediction partition, and upon determining the transform partitioning structure comprises splitting the parent transform unit into one or more non-square transforms, determining whether to split the one or more non-square transforms based at least in part on the one or more non-square transforms being non-square.