A security system processing method of a User Equipment (UE) and a security system for a wireless communication system are provided. The security processing method of the UE includes transmitting a Layer 3 message including a UE security capability to a Mobility Management Entity (MME) and the eNB, receiving a Access Stratum Security Mode Command (AS SMC) including a AS security algorithm selected by the eNB, as a result of verification of the UE security capability and information received from the MME, and a AS Message Authentication Code (MAC), transmitting a AS security mode complete message including the AS SMC to the eNB after verification of integrity of the AS SMC using the AS MAC, and transmitting, when receiving a Non Access Stratum (NAS) SMC including the UE security capability, a NAS security mode complete message to the MME after verification of integrity of the NAS SMC.