A method and apparatus of video encoding or decoding for a video encoding or decoding system applied to multi-face sequences corresponding to a 360-degree virtual reality sequence are disclosed. According the present invention, one or more multi-face sequences representing the 360-degree virtual reality sequence are derived. If Inter prediction is selected for a current block in a current face, one virtual reference frame is derived for each face of said one or more multi-face sequences by assigning one target reference face to a center of said one virtual reference frame and connecting neighboring faces of said one target reference face to said one target reference face at boundaries of said one target reference face. Then, the current block in the current face is encoded or decoded using a current virtual reference frame derived for the current face to derive an Inter predictor for the current block.