The present disclosure relates to a pre-5th-Generation (5G) or 5G communication system for supporting higher data rates Beyond 4th-Generation (4G) communication systems such as Long Term Evolution (LTE). A method for operating a receiver in a wireless communication system may include: receiving a signal from a transmitter; performing Integer Forcing (IF) equalization on the received signal; determining a log LikeLihood Ratio (LLR) value of each bit by using a posteriori probability of each bit for the signal determined based on an equalization matrix and a likelihood value for the signal; and decoding the signal by using the LLR value.