A method of sub-PU (prediction unit) syntax element signaling for a three-dimensional or multi-view video coding system is disclosed. A first syntax element associated with a texture sub-PU size is transmitted only for texture video data and a second syntax element associated with a depth sub-PU size is transmitted only for depth video data. The first syntax element associated with the texture sub-PU size is used to derive an IVMP (inter-view motion prediction) prediction candidate used for a texture block. The second syntax element associated with the depth sub-PU size is used to a MPI (motion parameter inheritance) prediction candidate for a depth block.