The present disclosure relates to a pre-5th-generation (5G) or 5G communication system to be provided for supporting higher data rates Beyond 4th-generation (4G) communication system, such as long-term evolution (LTE). An operation method of a transmitter in a wireless communication system is provided. The method includes applying a filter to data, mapping the data to which the filter is applied to at least one subcarrier, and transmitting the mapped data to a receiver. The filter is determined based on based on the allocated resource.