A method processes video data by intra prediction in a video coding system. In one implementation, the method receives input data associated with a current block in a current picture, wherein the current block is a non-square block and is coded or to be coded by intra prediction. The method then determines a first set of angular intra prediction modes for the longer side of the current block, wherein a number of the first set of the angular intra prediction modes is greater than a number of a second set of angular intra prediction modes for the shorter side of the current block. The method then determines a current intra prediction mode to derive an intra predictor for the current block, and encodes or decoding the current block according to the current intra prediction mode.