Disclosed herein is a method of performing an adaptive video coding, comprising: determining transform subsets including a group index and linear transforms with dimensions M×M and N×N, wherein the linear transforms correspond to at least one of a null transform and predefined transforms; selecting an optimal transform subset for a transform unit from the determined transform subsets, wherein each of rows and columns of the transform unit corresponds to different linear transform; and encoding the optimal transform subset.