公开(公告)号：US20250110940A1

公开(公告)日：2025-04-03

申请号：US18905090

申请日：2024-10-02

Applicant: Google LLC

Inventor： Xin Yang Yak ,  Sercan Omer Arik ,  Yihe Dong ,  Javier Gonzalvo Fructuoso

IPC: G06F16/22 ,  G06N3/08

Abstract: Methods, systems, and apparatuses, including computer programs encoded on computer storage media, for implementing a neural network that can perform one or more machine learning tasks on an input that includes data that represents a given data structure. In particular, implementing a language model to encode the data and a foundation neural network with an attention-based architecture to generate the task output. Because of how language model generated embeddings are defined and cached, the described techniques demonstrate significant improvements in required computational resources for training and inference while also exceeding prediction performance on a variety of prediction tasks over conventional approaches.

公开(公告)号：US20240119265A1

公开(公告)日：2024-04-11

申请号：US18373417

申请日：2023-09-27

Applicant: Google LLC

Inventor： Sercan Omer Arik ,  Yihe Dong ,  Qi Yu ,  Rui Wang

IPC: G06N3/0455 ,  G06N3/08

CPC classification number: G06N3/0455 ,  G06N3/08

Abstract: Aspects of the disclosure provide a deep sequence model, referred to as Koopman Neural Forecaster (KNF), for time series forecasting. KNF leverages deep neural networks (DNNs) to learn the linear Koopman space and the coefficients of chosen measurement functions. KNF imposes appropriate inductive biases for improved robustness against distributional shifts, employing both a global operator to learn shared characteristics, and a local operator to capture changing dynamics, as well as a specially-designed feedback loop to continuously update the learnt operators over time for rapidly varying behaviors. KNF achieves superior performance on multiple time series datasets that are shown to suffer from distribution shifts.

公开(公告)号：US20250111285A1

公开(公告)日：2025-04-03

申请号：US18902137

申请日：2024-09-30

Applicant: Google LLC

Inventor： Yan Liu ,  Chuizheng Meng ,  Yihe Dong ,  Sercan Omer Arik ,  Tomas Pfister

IPC: G06N20/00

Abstract: A machine-learned model includes an encoder having a feature block configured to embed input data into a plurality of features in an embedding space. The input data includes multiple components such as covariate, treatment, and output components. The encoder includes one or more encoding layers, each including a temporal attention block and a feature-wise attention block. The temporal attention block is configured to obtain the embedded input data and apply temporal causal attention along a time dimension in parallel for each feature of the plurality of features to generate temporal embeddings. The feature-wise attention block is configured to obtain the temporal embeddings and generate component representations such as a covariate representation, a treatment representation, and an output representation.

公开(公告)号：US20240386321A1

公开(公告)日：2024-11-21

申请号：US18639519

申请日：2024-04-18

Applicant: Google LLC

Inventor： Sayna Ebrahimi ,  Yihe Dong ,  Tomas Pfister ,  Sercan Omer Arik

IPC: G06N20/00

Abstract: Aspects of the disclosure are directed to a multimodal processing system for processing both structured and un-structured data. Real-world data is not always consistent in form or content. The multimodal processing system includes model that can be trained to account for this characteristic of real-world data, by selectively masking data of different modalities during pretraining to learn outputs that are the same or comparable between the masked and un-masked inputs. The model is trained according to modality-specific masking objectives computed for each modality of data and joint modality similarity-based masking objectives for a joint representation of the data across all modalities. The system provides consistent and accurate input, even when input data may have substantial portions of data from different modalities missing. Cross-modal relationships in data are reinforced by the model as different portions of data are masked, contributing to an overall increase in model accuracy versus other approaches.

公开(公告)号：US20240112084A1

公开(公告)日：2024-04-04

申请号：US18372900

申请日：2023-09-26

Applicant: Google LLC

Inventor： Sercan Omer Arik ,  Yihe Dong

IPC: G06N20/00

CPC classification number: G06N20/00

Abstract: Aspects of the disclosure are directed to a canonical approach for feature selection referred to as sparse learnable masks (SLM). SLM integrates learnable sparse masks into end-to-end training. For the fundamental non-differentiability challenge of selecting a desired number of features, SLM includes dual mechanisms for automatic mask scaling by achieving a desired feature sparsity and gradually tempering this sparsity for effective learning. SLM further employs an objective that increases mutual information (MI) between selected features and labels in an efficient and scalable manner. Empirically, SLM can achieve or improve upon state-of-the-art results on several benchmark datasets, often by a significant margin, while reducing computational complexity and cost.