A method and apparatus are provided. The method includes receiving a desired signal from a serving base station, receiving a plurality of interfering signals from one or more base stations, estimating a maximum likelihood (ML) decision metric of interfering signals, applying a logarithm function to the ML decision metric, and applying a maximum-log approximation function to a serving data vector and an interference data vector, which are included in the ML decision metric, determining the values of a transmit power, a rank, a precoding matrix, a modulation order and a transmission scheme using the applied ML decision metric, and cancelling the interfering signals from the received signals using the determined values of transmit power, rank, precoding matrix, modulation order and transmission scheme.