A signal detection apparatus and method using a modified stack algorithm in a Multi-Input Multi-Output (MIMO) system are provided. The signal detection method includes sorting signals received via antennas and channel coefficients for respective users in descending order, decomposing a channel matrix composed of the sorted channel coefficients into a unitary matrix and an upper-triangular matrix, determining the number of candidate symbol-sequences using the decomposed upper-triangular matrix, obtaining a signal vector for the antennas by using the sorted signals received via respective antennas and the unitary matrix, wherein the signal vector is proportional to the upper-triangular matrix, and detecting the determined number of candidate symbol-sequences by using a modified stack algorithm while expanding a stack structure for the obtained signal vector.