A method and apparatus for Intra prediction of a block based on neighboring pixels around the block are disclosed. Embodiments according to the present invention use square blocks as well as non-square blocks for Intra prediction. For a 2N×2N Luma CU (coding unit), the CU can be partitioned into 2N×N, N×2N, 2N×2N or N×N PUs. The 2N×N and N×2N PUs can be further processed by either square transforms only or both non-square and square transforms. In one embodiment, the 2N×N PU or the N×2N PU is processed as two N×N TUs (transform units) and each of the N×N TU is further split into smaller N×N TUs based on quad-tree split. In another embodiment, the 2N×N and N×2N PUs are processed as two 2N×0.5N and 0.5N×2N TUs respectively.