A method and system for determining a communication mode in a communication system are provided. The system and method include determining whether a first MS and a second MS are located within a preset area where P2P communications are available, estimating a first expected performance that can be achieved when the first and second MSs operate in a cellular communication mode, estimating a second expected performance that can be achieved when the first and second MSs operate in a P2P communication mode if the first and second MSs are located within the predetermined area, and selecting one of the cellular communication mode and the P2P communication mode as a communication mode between the first and second MSs by comparing the first expected performance with the second expected performance.