The present disclosure relates to a pre-5th-generation (5G) or 5G communication system to be provided for supporting higher data rates beyond 4th-generation (4G) communication system such as a long term evolution (LTE). An operating method of a transmitting apparatus in a wireless communication system is provided. The operating method includes detecting an achievable rate region for a receiving apparatus, and determining a modulation and coding scheme (MCS) level and a sliding window superposition coding (SWSC) multi-input multi-output (MIMO) scheme corresponding to the achievable rate region.