An apparatus and method for lowering a tree searching complexity in a Spatial Multiplexing (SM) Multiple Input Multiple Output (MIMO) antenna system are provided. A transmitting apparatus includes a generator for generating a plurality of transmit streams by spatial-multiplexing a transmit data, a determiner for determining modulation schemes to be applied to the plurality of the transmit streams respectively and a modulator for modulating the plurality of the transmit streams using at least two modulation schemes under control of the determiner. The receiver detects the signals by allocating the modulation scheme of the lowest order to the upper tree level.