A method of assigning a carrier for mitigating inter-cell interference in a fixed relay based communication system is provided. In a normal mode, the method includes defining at least one sector group by grouping cells at least two tiers away from each other; selecting a single sector group; and assigning, by a base station of the selected sector group, a first carrier to a relay station or a mobile station which is located in a coverage area of the base station. In a relay mode, the method includes determining whether a current mode is a normal mode or a relay mode; determining whether a base station is included in a selected sector group; and assigning carriers to a relay station or a mobile station belonging to the base station, when the current mode is the normal mode and the base station is included in the selected sector group.