An In-phase/Quadrature phase (I/Q) mismatch compensation method of a transceiver is provided. The method includes establishing a first loopback path between a transmitter and a receiver, transmitting a training sequence from the transmitter to the receiver through the established first loopback path, acquiring, at the receiver, a first correlation result value by correlating the training sequence received through the established first loopback path, establishing a second loopback path between the transmitter and the receiver, transmitting the training sequence from the transmitter to the receiver through the established second loopback path, acquiring, at the receiver, a second correlation result value by correlating the training sequence received through the second loopback path, and estimating I/Q mismatch values of the transmitter and the receiver using the first and second correlation result values.