A method and apparatus for decomposing a channel matrix in a wireless communication system are disclosed. A channel matrix H is generated for channels between transmit antennas and receive antennas. A Hermitian matrix A=HHH or A=HHH is created. A Jacobi process is cyclically performed on the matrix A to obtain Q and DA matrixes such that A=QDAQH. DA is a diagonal matrix obtained by singular value decomposition (SVD) on the A matrix. In each Jacobi transformation, real part diagonalization is performed to annihilate real parts of off-diagonal elements of the matrix and imaginary part diagonalization is performed to annihilate imaginary parts of off-diagonal elements of the matrix after the real part diagonalization. U, V and DH matrixes of H matrix are then calculated from the Q and DA matrices. DH is a diagonal matrix comprising singular values of the H matrix.