A communication system includes a transmitter that encodes binary bits of each of a plurality of data streams into a plurality of symbols and converts the plurality of symbols into a plurality of output signals, respectively corresponding to a plurality of channels, the converting based on a transmission rule defined by a first matrix; and a receiver that combines the plurality of output signals, received through the plurality of channels, the combining based on a reception rule defined by a second matrix, the combining restoring the plurality of symbols, and the receiver decodes the plurality of symbols into the binary bits. The first matrix and the second matrix are determined based on a third matrix that models a crosstalk effect between adjacent channels from among the plurality of channels, to reduce the crosstalk effect.