An apparatus and method for cooperative relay in a multiple-antenna wireless communication system based on relay stations (RSs) are provided. The apparatus includes a serial/parallel converter for dividing data into N number of streams, a channel estimator for calculating a first transmission rate at which a RS performing the cooperative relay can perform decoding and a second transmission rate at which all relay stations performing the cooperative relay can commonly perform decoding, and determining transmission rates for each of the N-number streams, and an adaptive modulation and coding (AMC) unit for encoding and modulating the N-number streams according to their respective transmission rates.