Provided are a system and a method for transmitting/receiving data in a communication system. The method includes calculating a processing matrix of a transmission unit for transmitting data, a processing matrix of a reception unit for receiving the data, a channel matrix formed by a plurality of transmission antennas and a plurality of reception antennas, and a parameter of the transmission data, when the data to be transmitted to the plurality of reception antennas via the plurality of transmission antennas is generated; determining a minimum value of a Minimum Mean Square Error (MMSE) according to the calculated matrixes and parameter; and transmitting/receiving the data using an MMSE multiplexing scheme based on the determined minimum MMSE value.