The present disclosure relates to a pre-5th-Generation (5G) or 5G communication system to be provided for supporting higher data rates Beyond 4th-Generation (4G) communication system such as Long Term Evolution (LTE). Exemplary embodiment of the present invention provide a scheme for reducing feedback overhead when quantizing and feeding back channel state information in a MIMO system. A method for operating a receiver in a MIMO system, according to one embodiment of the present invention, includes the steps of: performing trellis coded quantization for channel information by using a codebook selected from a plurality of codebooks; and transmitting, to a transmitter, feedback information including the quantization result. The step of performing trellis-coded quantization for the channel information includes a step of truncating the channel information and codewords included in the selected codebook into multiple groups of channel vectors and multiple groups of codewords, respectively, and performing trellis-coded quantization for each of the groups of the channel vectors by using each of the groups of the codewords.