Three block concepts are introduced in HEVC: coding unit (CU), prediction unit (PU), and transform unit (TU). The overall coding structure is characterized by the various sizes of CU, PU and TU in a recursive fashion. For transform processing in current HEVC, a hierarchy RQT (Residual Quad Tree) is used and the TU size is related to the CU size, but independent of the PU size. This results in high encoding complexity and also causes increased processing time to process the syntax of residual quad tree. Accordingly a modified transform unit partition with reduced complexity is disclosed. According to an embodiment, the TU size may be restricted to the minimum of PU width and height, except for a 2N×2N coding unit with the 2N×2N partition type. In another embodiment, the maximum TU size equals to maximum of PU width and height, and the minimum TU size equals to minimum of the PU width and height, except for a 2N×2N coding unit with the 2N×2N partition type. In yet another embodiment, the TU size is selected between 2N×2N and N×N for the 2N×2N, 2N×N, N×2N and N×N partition types. The syntax element, split_transform_flag, is used to indicate the selection of 2N×2N or N×N TU size when needed. Furthermore, a method with reduced complexity of selecting the best merge candidate for the 2N×2N CU merge mode is disclosed. The method relies on R-D cost associated with the motion vector of merge candidate to reduce required computation.