A method and apparatus for three-dimensional video coding or multi-view video coding are disclosed. Embodiments according to the present invention derive a unified disparity vector from depth information for Inter mode and Skip/Direct mode. The unified disparity vector is derived from a subset of depth samples in an associated depth block corresponding to the current block using a unified derivation method. The unified derivation method is applied in Inter mode, Skip mode, or Direct mode when a disparity vector derived from depth data is required for encoding or decoding. The unified disparity vector can also be applied to derive a disparity vector for locating a corresponding block, and thus an inter-view motion vector candidate can be determined for Skip mode or Direct mode.