公开(公告)号：US20240354504A1

公开(公告)日：2024-10-24

申请号：US18684557

申请日：2021-08-25

Applicant: Google LLC

Inventor： Chen-Yu Lee ,  Chun-Liang Li ,  Timothy Dozat ,  Vincent Perot ,  Guolong Su ,  Nan Hua ,  Joshua Ainslie ,  Renshen Wang ,  Yasuhisa Fujii ,  Tomas Pfister

IPC: G06F40/284 ,  G06V30/10 ,  G06V30/416

CPC classification number: G06F40/284 ,  G06V30/10 ,  G06V30/416

Abstract: Systems and methods for providing a structure-aware sequence model that can interpret a document's text without first inferring the proper reading order of the document. In some examples, the model may use a graph convolutional network to generate contextualized “supertoken” embeddings for each token, which are then fed to a transformer that employs a sparse attention paradigm in which attention weights for at least some supertokens are modified based on differences between predicted and actual values of the order and distance between the attender and attendee supertokens.