公开(公告)号：US11487832B2

公开(公告)日：2022-11-01

申请号：US16619777

申请日：2019-05-09

Applicant: Google LLC

Inventor： Gökhan Bakir ,  Andre Elisseeff ,  Torsten Marek ,  João Paulo Pagaime da Silva ,  Mathias Carlen ,  Dana Ritter ,  Lukasz Suder ,  Ernest Galbrun ,  Matthew Stokes ,  Marcin Nowak-Przygodzki ,  Mugurel-Ionut Andreica ,  Marius Dumitran

IPC: G06F16/00 ,  G06F16/9535 ,  G06F16/9032

Abstract: Implementations are described herein for analyzing existing interactive web sites to facilitate automatic engagement with those web sites, e.g., by automated assistants or via other user interfaces, with minimal effort from the hosts of those websites. For example, in various implementations, techniques described herein may be used to abstract, validate, maintain, generalize, extend and/or distribute individual actions and “traces” of actions that are useable to navigate through various interactive websites. Additionally, techniques are described herein for leveraging these actions and/or traces to automate aspects of interaction with a third party website. For example, in some implementations, techniques described herein may enable users to engage with an automated assistant (via a spoken or typed dialog session) to interact with the third party web site without requiring the user to visually interact with the third party web site directly and without requiring the third party to implement their own third party agent.

公开(公告)号：US11238868B2

公开(公告)日：2022-02-01

申请号：US16614224

申请日：2019-06-13

Applicant: Google LLC

Inventor： Denis Burakov ,  Behshad Behzadi ,  Mario Bertschler ,  Bohdan Vlasyuk ,  Daniel Cotting ,  Michael Golikov ,  Lucas Mirelmann ,  Steve Cheng ,  Sergey Nazarov ,  Zaheed Sabur ,  Marcin Nowak-Przygodzki ,  Mugurel Ionut Andreica ,  Radu Voroneanu

IPC: G10L15/22 ,  G10L15/26 ,  G06F3/16

Abstract: Implementations set forth herein relate to a system that employs an automated assistant to further interactions between a user and another application, which can provide the automated assistant with permission to initialize relevant application actions simultaneous to the user interacting with the other application. Furthermore, the system can allow the automated assistant to initialize actions of different applications, despite being actively operating a particular application. Available actions can be gleaned by the automated assistant using various application-specific schemas, which can be compared with incoming requests from a user to the automated assistant. Additional data, such as context and historical interactions, can also be used to rank and identify a suitable application action to be initialized via the automated assistant.

公开(公告)号：US20200342018A1

公开(公告)日：2020-10-29

申请号：US16617360

申请日：2018-05-07

Applicant: Google LLC

Inventor： Joseph Lange ,  Mugurel Ionut Andreica ,  Marcin Nowak-Przygodzki

IPC: G06F16/33 ,  G10L15/22 ,  G06F16/215 ,  G06F16/332 ,  G06F16/338

Abstract: Implementations are directed to determining, based on a submitted query that is a compound query, that a set of multiple sub-queries are collectively an appropriate interpretation of the compound query. Those implementations are further directed to providing, in response to such a determination, a corresponding command for each of the sub-queries of the determined set. Each of the commands is to a corresponding agent (of one or more agents), and causes the agent to generate and provide corresponding responsive content. Those implementations are further directed to causing content to be rendered in response to the submitted query, where the content is based on the corresponding responsive content received in response to the commands.

公开(公告)号：US20200250433A1

公开(公告)日：2020-08-06

申请号：US16850294

申请日：2020-04-16

Applicant: Google LLC

Inventor： Marcin Nowak-Przygodzki ,  Gökhan Bakir

IPC: G06K9/00 ,  G06F16/9032 ,  G06F3/01 ,  G06F3/03 ,  G06F3/00 ,  G06F9/451 ,  G06F16/58 ,  G06F3/0481 ,  G06F3/16 ,  H04N5/232

Abstract: Methods, apparatus, systems, and computer-readable media are set forth for generating and/or utilizing image shortcuts that cause one or more corresponding computer actions to be performed in response to determining that one or more features are present in image(s) from a camera of a computing device of a user (e.g., present in a real-time image feed from the camera). An image shortcut can be generated in response to user interface input, such as a spoken command. For example, the user interface input can direct the automated assistant to perform one or more actions in response to object(s) having certain feature(s) being present in a field of view of the camera. Subsequently, when the user directs their camera at object(s) having such feature(s), the assistant application can cause the action(s) to be automatically performed. For example, the assistant application can cause data to be presented and/or can control a remote device in accordance with the image shortcut.

公开(公告)号：US20240361982A1

公开(公告)日：2024-10-31

申请号：US18765101

申请日：2024-07-05

Applicant: GOOGLE LLC

Inventor： Joseph Lange ,  Marcin Nowak-Przygodzki

IPC: G06F3/16 ,  G10L15/22 ,  G10L15/28

CPC classification number: G06F3/167 ,  G10L15/22 ,  G10L15/28 ,  G10L2015/223

Abstract: Implementations set forth herein relate to an automated assistant that can provide a selectable action intent suggestion when a user is accessing a third party application that is controllable via the automated assistant. The action intent can be initialized by the user without explicitly invoking the automated assistant using, for example, an invocation phrase (e.g., “Assistant . . . ”). Rather, the user can initialize performance of the corresponding action by identifying one or more action parameters. In some implementations, the selectable suggestion can indicate that a microphone is active for the user to provide a spoken utterance that identifies a parameter(s). When the action intent is initialized in response to the spoken utterance from the user, the automated assistant can control the third party application according to the action intent and any identified parameter(s).

公开(公告)号：US20240274132A1

公开(公告)日：2024-08-15

申请号：US18642010

申请日：2024-04-22

Applicant: GOOGLE LLC

Inventor： Joseph Lange ,  Abhanshu Sharma ,  Adam Coimbra ,  Gökhan Bakir ,  Gabriel Taubman ,  IIya Firman ,  Jindong Chen ,  James Stout ,  Marcin Nowak-Przygodzki ,  Reed Enger ,  Thomas Weedon Hume ,  Vishwath Mohan ,  Jacek Szmigiel ,  Yunfan Jin ,  Kyle Pedersen ,  Gilles Baechler

IPC: G10L15/22 ,  G06F3/16 ,  G06F40/247 ,  G06F40/30 ,  G10L15/18

CPC classification number: G10L15/22 ,  G06F3/167 ,  G06F40/247 ,  G06F40/30 ,  G10L15/1815 ,  G10L15/1822 ,  G10L2015/223 ,  G10L2015/228

Abstract: Implementations set forth herein relate to an automated assistant that can interact with applications that may not have been pre-configured for interfacing with the automated assistant. The automated assistant can identify content of an application interface of the application to determine synonymous terms that a user may speak when commanding the automated assistant to perform certain tasks. Speech processing operations employed by the automated assistant can be biased towards these synonymous terms when the user is accessing an application interface of the application. In some implementations, the synonymous terms can be identified in a responsive language of the automated assistant when the content of the application interface is being rendered in a different language. This can allow the automated assistant to operate as an interface between the user and certain applications that may not be rendering content in a native language of the user.

公开(公告)号：US20230377572A1

公开(公告)日：2023-11-23

申请号：US18231112

申请日：2023-08-07

Applicant: GOOGLE LLC

Inventor： Mugurel-Ionut Andreica ,  Vladimir Vuskovic ,  Joseph Lange ,  Sharon Stovezky ,  Marcin Nowak-Przygodzki

IPC: G10L15/22 ,  G06N3/08 ,  G10L15/02

CPC classification number: G10L15/22 ,  G06N3/08 ,  G10L15/02 ,  G10L2015/223

Abstract: Implementations are set forth herein for creating an order of execution for actions that were requested by a user, via a spoken utterance to an automated assistant. The order of execution for the requested actions can be based on how each requested action can, or is predicted to, affect other requested actions. In some implementations, an order of execution for a series of actions can be determined based on an output of a machine learning model, such as a model that has been trained according to supervised learning. A particular order of execution can be selected to mitigate waste of processing, memory, and network resources—at least relative to other possible orders of execution. Using interaction data that characterizes past performances of automated assistants, certain orders of execution can be adapted over time, thereby allowing the automated assistant to learn from past interactions with one or more users.

公开(公告)号：US20230169102A1

公开(公告)日：2023-06-01

申请号：US18103291

申请日：2023-01-30

Applicant: Google LLC

Inventor： Joseph Lange ,  Mugurel Ionut Andreica ,  Marcin Nowak-Przygodzki

IPC: G06F16/33 ,  G06F16/215 ,  G06F16/332 ,  G06F16/338 ,  G10L15/22

CPC classification number: G06F16/3344 ,  G06F16/215 ,  G06F16/3329 ,  G06F16/338 ,  G10L15/22 ,  G10L15/26

Abstract: Implementations are directed to determining, based on a submitted query that is a compound query, that a set of multiple sub-queries are collectively an appropriate interpretation of the compound query. Those implementations are further directed to providing, in response to such a determination, a corresponding command for each of the sub-queries of the determined set. Each of the commands is to a corresponding agent (of one or more agents), and causes the agent to generate and provide corresponding responsive content. Those implementations are further directed to causing content to be rendered in response to the submitted query, where the content is based on the corresponding responsive content received in response to the commands.

公开(公告)号：US11600065B2

公开(公告)日：2023-03-07

申请号：US17838914

申请日：2022-06-13

Applicant: Google LLC

Inventor： Marcin Nowak-Przygodzki ,  Gökhan Bakir

IPC: G06F3/01 ,  G06F9/451 ,  G06F3/0481 ,  G06F3/16 ,  H04N5/232 ,  G06V20/20 ,  G06F16/9032 ,  G06F3/03 ,  G06F3/00 ,  G06F16/58

Abstract: Methods, apparatus, systems, and computer-readable media are set forth for generating and/or utilizing image shortcuts that cause one or more corresponding computer actions to be performed in response to determining that one or more features are present in image(s) from a camera of a computing device of a user (e.g., present in a real-time image feed from the camera). An image shortcut can be generated in response to user interface input, such as a spoken command. For example, the user interface input can direct the automated assistant to perform one or more actions in response to object(s) having certain feature(s) being present in a field of view of the camera. Subsequently, when the user directs their camera at object(s) having such feature(s), the assistant application can cause the action(s) to be automatically performed. For example, the assistant application can cause data to be presented and/or can control a remote device in accordance with the image shortcut.

公开(公告)号：US20230013581A1

公开(公告)日：2023-01-19

申请号：US17944712

申请日：2022-09-14

Applicant: GOOGLE LLC

Inventor： Marcin Nowak-Przygodzki ,  Jan Lamecki ,  Behshad Behzadi

IPC: H04L51/02 ,  H04L12/18 ,  G10L15/18 ,  G10L15/22 ,  G06Q10/10 ,  H04M3/493 ,  H04M3/527

Abstract: Techniques are described related to enabling automated assistants to enter into a “conference mode” in which they can “participate” in meetings between multiple human participants and perform various functions described herein. In various implementations, an automated assistant implemented at least in part on conference computing device(s) may be set to a conference mode in which the automated assistant performs speech-to-text processing on multiple distinct spoken utterances, provided by multiple meeting participants, without requiring explicit invocation prior to each utterance. The automated assistant may perform semantic processing on first text generated from the speech-to-text processing of one or more of the spoken utterances, and generate, based on the semantic processing, data that is pertinent to the first text. The data may be output to the participants at conference computing device(s). The automated assistant may later determine that the meeting has concluded, and may be set to a non-conference mode.