A method and apparatus of Inter prediction for video coding using UMVE (Ultimate Motion Vector Expression) are disclosed. According to this method, a base candidate is selected from an original Inter candidate list, where the base candidate has a base prediction direction. One or more extended candidates are derived by adding one or more offsets to the base candidate, where said one or more extended candidates have a same prediction direction as the base prediction direction. The extended candidates are then inserted into the original Inter candidate list to form an extended Inter candidate list for encoding or decoding. According to another method, the UMVE mode is treated as an additional AMVP (Advanced Motion Vector Prediction) mode.