The present disclosure relates to a pre-5th-Generation (5G) or 5G communication system to be provided for supporting higher data rates Beyond 4th-Generation (4G) communication system such as Long Term Evolution (LTE). A method for operating a receiving device in a wireless communication system comprises determining inter-symbol interference between symbols in a received signal, determining a location of a receive detection window according to the inter-symbol interference, and demodulating the received signal based on the location of the receive detection window. A receiving device includes at least one transceiver, and at least one processor configured to determine inter-symbol interference between symbols in a received signal, determine a location of a receive detection window according to the inter-symbol interference, and demodulate the received signal based on the location of the receive detection window. A transmitting device includes at least one processor configured to estimate an equivalent channel frequency response based on characteristic information of a time-domain filter, estimate an inter-symbol interference based on the equivalent channel frequency response, and generate indication information regarding an adjustment of a location of a receive detection window.