-
公开(公告)号:US20250061334A1
公开(公告)日:2025-02-20
申请号:US18805978
申请日:2024-08-15
Applicant: NEC Laboratories America, Inc.
Inventor: Yanchi Liu , Wei Cheng , Xujiang Zhao , Runxue Bao , Haifeng Chen , Nan Zhang
IPC: G06N3/082 , G06N3/0455
Abstract: Systems and methods for optimizing large language models (LLM) with domain-oriented model compression. Importance weights for general knowledge in a trained LLM, pretrained with deep learning, can be determined by computing the error when removing a weight from the trained LLM. The trained LLM can be iteratively optimized to obtain a domain-compressed LLM with domain knowledge while maintaining general knowledge by: fine-tuning the trained LLM iteratively with domain knowledge using the importance weights for general knowledge to obtain a fine-tuned LLM; determining importance weights for domain knowledge in the LLM with a regularization term by using gradient descent to optimize parameters when the fine-tuned LLM is trained with domain knowledge; and pruning learned knowledge based on importance weights for domain knowledge. A corrective action can be performed on a monitored entity using the domain-compressed LLM.