Invention Grant
- Patent Title: Token-position handling for sequence based neural networks
-
Application No.: US16848748Application Date: 2020-04-14
-
Publication No.: US11544537B2Publication Date: 2023-01-03
- Inventor: Andrew Wagner , Tiyasa Mitra , Sujeeth Subramanya Bharadwaj , Marc Tremblay , Saurabh Mohan Kulkarni
- Applicant: Microsoft Technology Licensing, LLC
- Applicant Address: US WA Redmond
- Assignee: Microsoft Technology Licensing, LLC
- Current Assignee: Microsoft Technology Licensing, LLC
- Current Assignee Address: US WA Redmond
- Agency: Fountainhead Law Group PC
- Main IPC: G06F40/289
- IPC: G06F40/289 ; G06N3/08 ; G06N3/04 ; G06K9/62

Abstract:
Embodiments of the present disclosure include a method for token-position handling comprising: processing a first sequence of tokens to produce a second sequence of tokens, wherein the second sequence of tokens has a smaller number of tokens than the first sequence of tokens; masking at least some tokens in the second sequence to produce masked tokens; moving the masked tokens to the beginning of the second sequence to produce a third sequence; encoding tokens in the third sequence into a set of numeric vectors in a first array; and processing the first array in a transformer neural network to determine correlations among the third sequence, the processing the first array producing a second array.
Public/Granted literature
- US20210319288A1 TOKEN-POSITION HANDLING FOR SEQUENCE BASED NEURAL NETWORKS Public/Granted day:2021-10-14
Information query