A method for generating a decoding metric in a wireless communication system includes receiving an FQAM symbol and determining transmission probabilities of FQAM symbol candidates from a received value of the FQAM symbol, wherein the transmission probabilities of the FQAM symbol candidates are determined by multiplying a probability that a QAM symbol included in the FQAM symbol is mapped to each of candidate values in one frequency and a probability that a signal is not transmitted in the other at least one frequency. An apparatus in a wireless communication system for supporting FQAM, the apparatus comprising an RF processing unit configured to receive an FQAM symbol, and a modem configured to determine transmission probabilities of FQAM symbol candidates from a received value of the FQAM symbol.