A method of operating a base station in a wireless communication system supporting Frequency-Quadrature Amplitude Modulation (FQAM) and Multi-Tone FQAM (MT-FQAM) is provided. The method includes determining a modulation scheme of data to be transmitted, and modulating the data according to the determined modulation scheme, wherein if at least one resource block is included in the data and if the at least one resource block is mapped to at least one tone in a distributed manner, the MT-FQAM scheme is selected, or if one resource block is included in the data and if the one resource block is mapped to at least one tone in a continuous manner, the FQAM scheme is selected, or if multiple resource blocks are included in the data and if the multiple resource blocks are mapped to at least one tone in a continuous manner, the MT-FQAM scheme is selected.