A method and apparatus for video coding using template-based Intra prediction are disclosed. According to one method, the template-based Intra prediction searches N template matching candidates to reduce the complexity. In another method, the template-based Intra prediction assigns different weights for different pixel locations of template or uses different pixel precision for interpolation filter during cost evaluation among template matching candidates. In yet another method, truncated template size is used to reduce the complexity. In yet another method, the left and top templates may have different sizes for non-square block. In yet another method, inverse templated-based Intra prediction is used.