公开(公告)号：US20250139379A1

公开(公告)日：2025-05-01

申请号：US18385270

申请日：2023-10-30

Applicant: GOOGLE LLC

Inventor： Sanil Jain ,  Wei Yu ,  Alessandro Agostini ,  Agoston Weisz ,  Michael Andrew Goodman ,  Attila Dankovics ,  Elle Chae ,  Evgeny Sluzhaev ,  Amin Ghafouri ,  Golnaz Ghiasi ,  Igor Petrovski ,  Konstantin Shagin ,  Marcelo Menegali ,  Oscar Akerlund ,  Rakesh Shivanna ,  Thang Luong ,  Tiffany Chen ,  Vikas Peswani ,  Yifeng Lu

IPC: G06F40/40 ,  G06F16/483

Abstract: Implementations relate to generating multi-modal response(s) through utilization of large language model(s) (LLM(s)) and other generative model(s). Processor(s) of a system can: receive natural language (NL) based input, generate a multi-modal response that is responsive to the NL based output, and cause the multi-modal response to be rendered. In some implementations, and in generating the multi-modal response, the processor(s) can process, using a LLM, LLM input to generate LLM output, and determine, based on the LLM output, textual content and generative multimedia content for inclusion in the multi-modal response. In some implementations, the generative multimedia content can be generated by another generative model (e.g., an image generator, a video generator, an audio generator, etc.) based on generative multimedia content prompt(s) included in the LLM output and that is indicative of the generative multimedia content. In various implementations, the generative multimedia content can be interleaved between segments of the textual content.

公开(公告)号：US12277400B1

公开(公告)日：2025-04-15

申请号：US18590498

申请日：2024-02-28

Applicant: GOOGLE LLC

Inventor： Sanil Jain ,  Wei Yu ,  Ágoston Weisz ,  Michael Andrew Goodman ,  Diana Avram ,  Amin Ghafouri ,  Golnaz Ghiasi ,  Igor Petrovski ,  Khyatti Gupta ,  Oscar Akerlund ,  Evgeny Sluzhaev ,  Rakesh Shivanna ,  Thang Luong ,  Komal Singh ,  Yifeng Lu ,  Vikas Peswani

IPC: G06F40/40 ,  G06V10/70

Abstract: Implementations relate to managing multimedia content that is obtained by large language model(s) (LLM(s)) and/or generated by other generative model(s). Processor(s) of a system can: receive natural language (NL) based input that requests multimedia content, generate a response that is responsive to the NL based input, and cause the response to be rendered. In some implementations, and in generating the response, the processor(s) can process, using a LLM, LLM input to generate LLM output, and determine, based on the LLM output, at least multimedia content to be included in the response. Further, the processor(s) can evaluate the multimedia content to determine whether it should be included in the response. In response to determining that the multimedia content should not be included in the response, the processor(s) can cause the response, including alternative multimedia content or other textual content, to be rendered.

公开(公告)号：US11682191B2

公开(公告)日：2023-06-20

申请号：US17702438

申请日：2022-03-23

Applicant: Google LLC

Inventor： Jon Shlens ,  Ekin Dogus Cubuk ,  Quoc Le ,  Tsung-Yi Lin ,  Barret Zoph ,  Golnaz Ghiasi

IPC: G06V10/772 ,  G06T3/60 ,  G06T3/20 ,  G06T11/00 ,  G06F18/24 ,  G06F18/21

CPC classification number: G06V10/772 ,  G06F18/217 ,  G06F18/24 ,  G06T3/20 ,  G06T3/60 ,  G06T11/001

Abstract: Example aspects of the present disclosure are directed to systems and methods for learning data augmentation strategies for improved object detection model performance. In particular, example aspects of the present disclosure are directed to iterative reinforcement learning approaches in which, at each of a plurality of iterations, a controller model selects a series of one or more augmentation operations to be applied to training images to generate augmented images. For example, the controller model can select the augmentation operations from a defined search space of available operations which can, for example, include operations that augment the training image without modification of the locations of a target object and corresponding bounding shape within the image and/or operations that do modify the locations of the target object and bounding shape within the training image.

公开(公告)号：US20190354817A1

公开(公告)日：2019-11-21

申请号：US16416848

申请日：2019-05-20

Applicant: Google LLC

Inventor： Jon Shlens ,  Ekin Dogus Cubuk ,  Quoc Le ,  Tsung-Yi Lin ,  Barret Zoph ,  Golnaz Ghiasi

IPC: G06K9/66 ,  G06K9/62 ,  G06T11/00 ,  G06T3/20 ,  G06T3/60

Abstract: Example aspects of the present disclosure are directed to systems and methods for learning data augmentation strategies for improved object detection model performance. In particular, example aspects of the present disclosure are directed to iterative reinforcement learning approaches in which, at each of a plurality of iterations, a controller model selects a series of one or more augmentation operations to be applied to training images to generate augmented images. For example, the controller model can select the augmentation operations from a defined search space of available operations which can, for example, include operations that augment the training image without modification of the locations of a target object and corresponding bounding shape within the image and/or operations that do modify the locations of the target object and bounding shape within the training image.

公开(公告)号：US20220301298A1

公开(公告)日：2022-09-22

申请号：US17697750

申请日：2022-03-17

Applicant: Google LLC

Inventor： Tsung-Yi Lin ,  Barret Zoph ,  Ekin Dogus Cubuk ,  Golnaz Ghiasi ,  Quoc V. Le

IPC: G06V10/82 ,  G06N3/08 ,  G06V10/774 ,  G06V10/77 ,  G06V10/776 ,  G06V10/764 ,  G06V10/80

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an image representation neural network.

公开(公告)号：US11301733B2

公开(公告)日：2022-04-12

申请号：US16416848

申请日：2019-05-20

Applicant: Google LLC

Inventor： Jon Shlens ,  Ekin Dogus Cubuk ,  Quoc Le ,  Tsung-Yi Lin ,  Barret Zoph ,  Golnaz Ghiasi

IPC: G06K9/66 ,  G06K9/62 ,  G06T3/60 ,  G06T3/20 ,  G06T11/00 ,  G06K9/00

Abstract: Example aspects of the present disclosure are directed to systems and methods for learning data augmentation strategies for improved object detection model performance. In particular, example aspects of the present disclosure are directed to iterative reinforcement learning approaches in which, at each of a plurality of iterations, a controller model selects a series of one or more augmentation operations to be applied to training images to generate augmented images. For example, the controller model can select the augmentation operations from a defined search space of available operations which can, for example, include operations that augment the training image without modification of the locations of a target object and corresponding bounding shape within the image and/or operations that do modify the locations of the target object and bounding shape within the training image.

公开(公告)号：US20220108204A1

公开(公告)日：2022-04-07

申请号：US17061355

申请日：2020-10-01

Applicant: Google LLC

Inventor： Xianzhi Du ,  Yin Cui ,  Tsung-Yi Lin ,  Quoc V. Le ,  Pengchong Jin ,  Mingxing Tan ,  Golnaz Ghiasi ,  Xiaodan Song

IPC: G06N20/00 ,  G06N3/04 ,  G06F11/34

Abstract: A computer-implemented method of generating scale-permuted models can generate models having improved accuracy and reduced evaluation computational requirements. The method can include defining, by a computing system including one or more computing devices, a search space including a plurality of candidate permutations of a plurality of candidate feature blocks, each of the plurality of candidate feature blocks having a respective scale. The method can include performing, by the computing system, a plurality of search iterations by a search algorithm to select a scale-permuted model from the search space, the scale-permuted model based at least in part on a candidate permutation of the plurality of candidate permutations.

公开(公告)号：US20220092387A1

公开(公告)日：2022-03-24

申请号：US17433677

申请日：2020-02-25

Applicant: Google LLC

Inventor： Quoc V. Le ,  Golnaz Ghiasi ,  Tsung-Yi Lin

IPC: G06N3/04

Abstract: A computing system for producing an architecture of a pyramid layer is disclosed. The computing system can include a controller model configured to generate new architectures for a pyramid layer that receives a plurality of input feature representations output by a backbone model and, in response, outputs a plurality of output feature representations. The plurality of input feature representations can have a plurality of different input resolutions, and the plurality of output feature representations can have a plurality of different output resolutions. The computing system can be configured to perform a plurality of iterations. For each iteration, the computing system can receive a new pyramid layer architecture as an output of the controller model and evaluate one or more performance characteristics of a machine-learned pyramidal feature model that includes the backbone model and one or more pyramid layers that have the new pyramid layer architecture.

公开(公告)号：US11907674B1

公开(公告)日：2024-02-20

申请号：US18370683

申请日：2023-09-20

Applicant: GOOGLE LLC

Inventor： Oscar Akerlund ,  Evgeny Sluzhaev ,  Golnaz Ghiasi ,  Thang Luong ,  Yifeng Lu ,  Igor Petrovski ,  Ágoston Weisz ,  Wei Yu ,  Rakesh Shivanna ,  Michael Andrew Goodman ,  Apoorv Kulshreshtha ,  Yu Du ,  Amin Ghafouri ,  Sanil Jain ,  Dustin Tran ,  Vikas Peswani ,  YaGuang Li

IPC: G06F40/00 ,  G06F40/40

CPC classification number: G06F40/40

Abstract: Implementations relate to generating multi-modal response(s) through utilization of large language model(s) (LLM(s)). Processor(s) of a system can: receive natural language (NL) based input, generate a multi-modal response that is responsive to the NL based output, and cause the multi-modal response to be rendered. In some implementations, and in generating the multi-modal response, the processor(s) can process, using a LLM, LLM input (e.g., that includes at least the NL based input) to generate LLM output, and determine, based on the LLM output, textual content for inclusion in the multi-modal response and multimedia content for inclusion in the multi-modal response. In some implementations, the multimedia content can be obtained based on a multimedia content tag that is included in the LLM output and that is indicative of the multimedia content. In various implementations, the multimedia content can be interleaved between segments of the textual content.

公开(公告)号：US20230274532A1

公开(公告)日：2023-08-31

申请号：US18313772

申请日：2023-05-08

Applicant: Google LLC

Inventor： Jon Shlens ,  Ekin Dogus Cubuk ,  Quoc Le ,  Tsung-Yi Lin ,  Barret Zoph ,  Golnaz Ghiasi

IPC: G06V10/772 ,  G06T3/60 ,  G06T3/20 ,  G06T11/00 ,  G06F18/24 ,  G06F18/21

CPC classification number: G06V10/772 ,  G06F18/217 ,  G06F18/24 ,  G06T3/20 ,  G06T3/60 ,  G06T11/001

Abstract: Example aspects of the present disclosure are directed to systems and methods for learning data augmentation strategies for improved object detection model performance. In particular, example aspects of the present disclosure are directed to iterative reinforcement learning approaches in which, at each of a plurality of iterations, a controller model selects a series of one or more augmentation operations to be applied to training images to generate augmented images. For example, the controller model can select the augmentation operations from a defined search space of available operations which can, for example, include operations that augment the training image without modification of the locations of a target object and corresponding bounding shape within the image and/or operations that do modify the locations of the target object and bounding shape within the training image.