Disclosed is a system and a method for controlling a handover of a mobile subscriber station in a broadband wireless access communication system. When serving base station determines to perform a handover of the mobile subscriber station, neighbor base stations to which the mobile subscriber station can be handed over, from among neighbor base stations of the mobile subscriber station, are determined based on a service quality level of service being currently provided to the mobile subscriber station. A handover request signal containing information related to the determined neighbor base station to which the mobile subscriber station can be handed over is transmitted to the mobile subscriber station. The serving base station receives from the mobile subscriber station in response to the request of the handover information about one neighbor base station to which the mobile subscriber station will be handed over, and informs the neighbor base station that the mobile subscriber station will be handed over to the neighbor base station.