The present invention relates to an activity recognition system balanced between versatility and individuation, comprising a communication framework jointly formed by a data collecting terminal, a computing device, and a cloud computing platform, the activity recognition system uses the communication framework to conduct personnel activity recognition and model updating, and the edge computing device further comprises a model training module and an activity recognition module, and the model training module retrieves a local activity recognition model by continuously verifying user IDs, and uses the first data to train a versatile network structure and an individualized network structure of the local activity recognition model in a way that individuation features of the user and versatility features of the model are fused with each other, so that the personnel activity recognition process conducted by the activity recognition module.