Method and apparatus of video coding using generalized bi-prediction are disclosed. According to one method, the generalized bi-prediction is extended for Decoder-Side Motion Vector Refinement (DMVR), where unequal weighting factor pair is allowed to form the template of the current block or the final predictor. In another method, the generalized bi-prediction is extended to pattern-based MV derivation (PMVD), where unequal weighting factor pair is allowed to combined reference blocks pointed by motion vectors derived based on PMVD. In yet another method, the generalized bi-prediction is used for Merge mode. When the final Merge MVP selected corresponds to bi-prediction using an unequal weighting factor pair, a new weighting factor pair is derived using neighboring reconstructed pixels of the current block and corresponding motion compensated pixels. In yet another method, the size of a set of weighting factor pairs is dependent on block size.