SMALL AND FAST TRANSFORMER MODEL FOR MULTI-MODAL OR OTHER TASKS

    公开(公告)号:US20230177338A1

    公开(公告)日:2023-06-08

    申请号:US18073383

    申请日:2022-12-01

    CPC classification number: G06N3/082 G06V10/82 G06V10/772

    Abstract: A method includes obtaining, using a first electronic device, a weight matrix associated with a trained transformer model. The method also includes factorizing the weight matrix into a dictionary weight matrix and an intermediate matrix. The method further includes pruning the intermediate matrix to generate a sparse intermediate matrix. The method also includes fine-tuning the sparse intermediate matrix based on a training dataset to generate a fine-tuned sparse intermediate matrix. The method further includes determining an index matrix and a coefficient matrix based on the fine-tuned sparse intermediate matrix. In addition, the method includes deploying the dictionary weight matrix, the index matrix, and the coefficient matrix to a second electronic device without deploying the weight matrix to the second electronic device. A number of parameters in the dictionary weight matrix, the index matrix, and the coefficient matrix is smaller than a number of parameters in the weight matrix.

Patent Agency Ranking