In a multiple input/multiple output wireless communication system, a receiving signal is compensated based on a wireless channel state information matrix converted by lattice reduction. The distortion of the modulation constellation diagram of a signal is considered, a detection signal is processed, and an intermediate signal table containing at least two intermediate signals is obtained to increase the area for the detection signal to be decided correctly, thereby to improve the possibility of the correct detection of the signal. Then a candidate signal table is obtained from the intermediate signal table and, based on the candidate signal table, a decision signal for a transmitting signal is obtained, thereby improving the accuracy of the detection of the transmitting signal. When at a high signal-to-noise ratio, the bit-error performance approaches the bit-error performance of a system using maximum likelihood detection algorithm, while causing no significant increase in computation complexity.