A method of signaling depth-based block partitioning (DBBP) for multi-view or three-dimensional (3D) video coding is disclosed. In one embodiment, the DBBP flag is signaled for all candidate prediction modes of the current texture coding unit including a non-2N×N partition mode. The group of candidate prediction modes may consist of 2N×N partition mode and N×2N partition mode. If the DBBP flag indicates the DBBP being used for the current texture coding unit, DBBP encoding is applied to the current texture coding unit or DBBP decoding is applied to one or more PUs associated with the current texture coding unit to recover the current texture coding unit. If the DBBP flag indicates the DBBP being not used for the current texture coding unit, a prediction partition mode is signaled at an encoder side or parsed at a decoder side from a CU (coding unit) level of the bitstream.