Multi-view video encoding using early Merge mode decision and/or early CU split termination is disclosed. The present invention encodes a current coding block using the Merge/Skip mode without evaluating coding performance for at least one of Inter modes and Intra modes if the coding conditions associated with the current coding block and a neighboring block set of the current coding block are asserted. The coding conditions may correspond to whether the coding performance of the Skip mode is better than the coding performance of 2N×2N Merge mode for the current coding block and whether all blocks in the neighboring block set select the Merge mode or the Skip mode. Similarly, the process of splitting the current coding block into smaller coding blocks can be terminated without evaluating coding performance associated with the smaller coding blocks from splitting the current coding block if some coding conditions are asserted.