A method and apparatus of three-dimensional/multi-view coding using aligned reference information are disclosed. The method operates by receiving input data associated with a current block of a current frame in a dependent view, determining a first DV (Disparity Vector) from one or more neighboring blocks of the current block, wherein the first DV refers to a first reference view to derive first reference information, selecting a second reference view for the current block to derive second reference information, aligning the first reference information associated with the first reference view with the second reference information associated with the second reference view, and applying inter-view encoding or decoding to the input data utilizing the first DV or the second reference information after applying said aligning the first reference information with the second reference information.