A method for selecting a Multiple-Input Multiple-Output (MIMO) mode based on energy efficiency and selecting an antenna subset to be used in a communication, in a wireless communication system using a multi-user massive multi-antenna is provided. The method selects an antenna subset capable of minimizing transmission power without using all antennas, based on power consumed by an Radio Frequency (RF) circuit as well as the transmission power. The method further includes selecting a mobile station to which the signal is to be transmitted, selecting a multi-antenna technique based on power consumption of all antennas, selecting an antenna subset to transmit the signal to the mobile station among all the antennas, and transmitting the signal to the mobile station by using the antenna subset.