Disclosed are a subscriber station authentication method, a protocol configuration method, and a device thereof in a wireless portable Internet system. In the subscriber station authentication method, an authentication mode between a subscriber station (10) and a base station (20) is negotiated, and the authentication mode is negotiated by the base station (20) according to the authentication mode negotiation. The authentication modes include an authentication mode based on the IEEE 802.16 privacy standard protocol and an authentication mode based on the standardized authentication protocol of the upper layer. Authentication is performed by the base station (10) in the case of the authentication mode based on the IEEE 802.16 privacy standard protocol, and the authentication is performed through message transmission using a diameter protocol between a base station (10) and an authentication server (40) in the case of the authentication mode based on the standardized authentication protocol of the upper layer.