Provided is a communication method of a secondary transmitter included in a multiple-antenna based multiple input multiple output (MIMO) cognitive radio communication system, the method including: estimating an interference channel between the secondary transmitter and a primary user in each of at least one first spectrum band that is being used in a primary system; calculating an orthogonal projection matrix to cancel interference with respect to the primary user from the interference channel; calculating an effective channel between the secondary transmitter and a secondary receiver based on the to orthogonal projection matrix; and selecting an index of an optimal spectrum band from the at least one first spectrum band based on the orthogonal projection matrix and the effective channel.