A method and apparatus for three-dimensional video encoding or decoding using the disparity vector derived from an associated depth block are disclosed. The method determines an associated depth block for a current texture block and derives a derived disparity vector based on a subset of depth samples of the associated depth block. The subset contains less depth samples than the associated depth block and the subset excludes a single-sample subset corresponding to a center sample of the associated depth block. The derived disparity vector can be used as an inter-view motion (disparity) vector predictor in Inter mode, an inter-view (disparity) candidate in Merge mode or Skip mode. The derived disparity vector can also be used to locate a reference block for inter-view motion prediction in Inter mode, inter-view candidate in Merge or Skip mode, inter-view motion prediction, inter-view disparity prediction, or inter-view residual prediction.