An apparatus and a method for operating a transmission end in a wireless communication system that supports Frequency and Quadrature-Amplitude Modulation (FQAM) are provided. The method includes dividing an information bit stream into a plurality of portions, encoding each of the plurality of portions using different encoding schemes, and generating an FQAM symbol by combining result values of the encoding of each of the plurality of portions, wherein the encoding schemes are different according to at least one of an encoding order, an encoding rate, an input size, and an encoding technique.