A method and apparatus for a three-dimensional or multi-view video encoding or decoding system are disclosed, where a three-dimensional coding tool relying on a disparity vector are adaptively applied depending on whether the inter-view reference picture pointed by the disparity vector is in the reference list associated with the current slice. The three-dimensional coding tool may correspond to the Inter-View Motion Prediction (IVMP) or View Synthesis Prediction (VSP). If the inter-view reference picture pointed by the DV is not in the current reference list associated with the current slice, the selected three-dimensional coding tool is disabled for the current block. If the inter-view reference picture pointed by the DV is in the current reference list associated with the current slice, the selected three-dimensional coding tool can be applied to the current block.