Method and apparatus of video coding using decoder derived motion information based on bilateral matching or template matching are disclosed. According to one method, an initial motion vector (MV) index is signalled in a video bitstream at an encoder side or determined from the video bitstream at a decoder side. A selected MV is then derived using bilateral matching, template matching or both to refine an initial MV associated with the initial MV index. In another method, when both MVs for list 0 and list 1 exist in template matching, the smallest-cost MV between the two MVs may be used for uni-prediction template matching if the cost is lower than the bi-prediction template matching. According to yet another method, the refinement of the MV search is dependent on the block size. According to yet another method, merge candidate MV pair is always used for bilateral matching or template matching.