An apparatus and method for detecting a signal in a receiver by maximum likelihood (ML) are provided, in which symbols are detected according to the number of transmit antennas of a transmitter and a modulation scheme, channels are estimated, an equivalent channel matrix corresponding to the estimated channels is determined, a permuted equivalent channel matrix is determined by multiplying the equivalent channel matrix by a predetermined permutation matrix, the permuted equivalent channel matrix is QR decomposed, a hard decision is performed on predetermined symbols among the detected symbols using a received signal resulting from the QR decomposition, and the log likelihood ratios (LLRs) of the hard-decided symbols are determined.