Receiving apparatus and method in a Multiple-Input Multiple-Output (MIMO) wireless communication system are provided. The receiver having N-ary receive antennas includes a decomposer for decomposing a channel matrix to a matrix Q and a matrix R through a QR decomposition; a detector for determining a candidate group of an n-th phase by estimating a plurality of transmit signal vectors by substituting a plurality of transmittable symbols into symbol combinations of a candidate group of a (n−1)-th phase as an n-th symbol and detecting (n+1)-th through N-th symbols using characteristics of the matrix R; a calculator for calculating square Euclidean distance values between the transmit signal vectors and a receive signal vector; and a determiner for determining the candidate group of the n-th phase by selecting transmit signal vectors having the smallest square Euclidean distance value among the transmit signal vectors.