公开(公告)号：US11734926B2

公开(公告)日：2023-08-22

申请号：US17093880

申请日：2020-11-10

Applicant: Google LLC

Inventor： Ibrahim Badr ,  Nils Grimsmo ,  Gökhan Bakir

IPC: G06K9/00 ,  G06V20/20 ,  G06F3/16 ,  G06F16/9032 ,  G06F16/583 ,  H04L51/02 ,  G06V20/68

CPC classification number: G06V20/20 ,  G06F3/167 ,  G06F16/5854 ,  G06F16/90332 ,  H04L51/02 ,  G06V20/68

Abstract: Methods, apparatus, and computer readable media are described related to causing processing of sensor data to be performed in response to determining a request related to an environmental object that is likely captured by the sensor data. Some implementations further relate to determining whether the request is resolvable based on the processing of the sensor data. When it is determined that the request is not resolvable, a prompt is determined and provided as user interface output, where the prompt provides guidance on further input that will enable the request to be resolved. In those implementations, the further input (e.g., additional sensor data and/or the user interface input) received in response to the prompt can then be utilized to resolve the request.

公开(公告)号：US11526570B2

公开(公告)日：2022-12-13

申请号：US17138612

申请日：2020-12-30

Applicant: Google LLC

Inventor： Ibrahim Badr

IPC: G06F16/954 ,  G06F16/957 ,  G06F16/9535 ,  H04L67/50

Abstract: Techniques are described herein for determining a predicted intent of a user and displaying additional content selected based on the predicted intent of the user. A method includes: receiving information identifying a webpage that a user is visiting and a navigational path of the user in navigating to the webpage; determining a predicted intent of the user based on the information identifying the webpage that the user is visiting and the navigational path of the user in navigating to the webpage; selecting additional content based upon the predicted intent of the user; and displaying an overlay, on a portion of the webpage, that includes the additional content.

公开(公告)号：US11442983B2

公开(公告)日：2022-09-13

申请号：US16731786

申请日：2019-12-31

Applicant: Google LLC

Inventor： Ibrahim Badr ,  Nils Grimsmo ,  Gokhan H. Bakir ,  Kamil Anikiej ,  Aayush Kumar ,  Viacheslav Kuznetsov

IPC: G06F16/58 ,  G06F16/9032 ,  G06V10/70 ,  G06V20/62 ,  G06V30/10

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for contextually disambiguating queries are disclosed. In an aspect, a method includes receiving an image being presented on a display of a computing device and a transcription of an utterance spoken by a user of the computing device, identifying a particular sub-image that is included in the image, and based on performing image recognition on the particular sub-image, determining one or more first labels that indicate a context of the particular sub-image. The method also includes, based on performing text recognition on a portion of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image, based on the transcription, the first labels, and the second labels, generating a search query, and providing, for output, the search query.

公开(公告)号：US11425071B2

公开(公告)日：2022-08-23

申请号：US17120927

申请日：2020-12-14

Applicant: Google LLC

Inventor： Ibrahim Badr ,  Paige Alexis Dunn-Rankin

IPC: H04L51/10 ,  G06F16/955 ,  G06F16/583 ,  H04L51/18 ,  H04L51/046 ,  H04L12/46 ,  G06T1/00 ,  G06F9/445

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating, in response to a single input operating system command that is invoked independent of a native application, a message that includes an image of a particular environment instance of the native application that was displayed when the single input operating system command and a uniform resource identifier of the particular environment instance of the native application.

公开(公告)号：US11169668B2

公开(公告)日：2021-11-09

申请号：US15981536

申请日：2018-05-16

Applicant: Google LLC

Inventor： Ibrahim Badr

IPC: G06F3/16 ,  G06F3/0484 ,  G06F3/0481 ,  H04M1/72454 ,  H04M1/72457

Abstract: Methods, systems, and apparatus for selecting an input mode are described. In one aspect, a method includes receiving request data specifying a request to launch a virtual assistant application from a lock screen of a mobile device. In response to receiving the request data, input signals are obtained. A selection of an input mode for the virtual assistant application is made, from candidate input modes, based on the input signals. Each candidate input mode is of an input type different from each other input type of each other candidate input mode. The input types include an image type and an audio type. The input mode of the image type receives pixel data for input to the virtual assistant application. The input mode of the audio type receives audio input for the virtual assistant application. The virtual assistant application presents content selected based on input signals received using the selected input mode.

公开(公告)号：US11086493B2

公开(公告)日：2021-08-10

申请号：US16905245

申请日：2020-06-18

Applicant: Google LLC

Inventor： Ibrahim Badr ,  Gokhan H. Bakir ,  Roland Peter Kehl ,  Nils Grimsmo

IPC: G06F3/0484 ,  G06F9/451 ,  G06F3/0482 ,  G06F3/16 ,  G06K9/00 ,  G08C17/02 ,  G06T19/00 ,  H04W88/02

Abstract: Methods, systems, and apparatus for controlling smart devices are described. In one aspect a method includes receiving image data for an image captured by a camera of a mobile device of a user and determining that the image depicts at least one of a smart device or a physical control for the smart device. In response to determining that that the image depicts a smart device or a physical control for the smart device, identifying one or more user interface controls for controlling the smart device, and generating and presenting, at a display of the mobile device, the one or more user interface controls for controlling the smart device. The method can further include detecting, at the display of the mobile device, user interaction with at least one of the one or more user interface controls, and controlling the smart device based on the detected user interaction.

公开(公告)号：US20200319765A1

公开(公告)日：2020-10-08

申请号：US16905245

申请日：2020-06-18

Applicant: Google LLC

Inventor： Ibrahim Badr ,  Gokhan H. Bakir ,  Roland Peter Kehl ,  Nils Grimsmo

IPC: G06F3/0484 ,  G06F9/451 ,  G06F3/0482 ,  G06F3/16 ,  G06K9/00 ,  G08C17/02

Abstract: Methods, systems, and apparatus for controlling smart devices are described. In one aspect a method includes receiving image data for an image captured by a camera of a mobile device of a user and determining that the image depicts at least one of a smart device or a physical control for the smart device. In response to determining that that the image depicts a smart device or a physical control for the smart device, identifying one or more user interface controls for controlling the smart device, and generating and presenting, at a display of the mobile device, the one or more user interface controls for controlling the smart device. The method can further include detecting, at the display of the mobile device, user interaction with at least one of the one or more user interface controls, and controlling the smart device based on the detected user interaction.

公开(公告)号：US20200293787A1

公开(公告)日：2020-09-17

申请号：US16891465

申请日：2020-06-03

Applicant: Google LLC

Inventor： Ibrahim Badr

IPC: G06K9/00 ,  H04N21/462 ,  G06F16/432 ,  G06F16/951 ,  G06F16/783 ,  G06F16/683 ,  G06K9/72

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing contextual information for presented media. In one aspect, a method includes storing in a buffer, on a first user device, media data as buffered media data, the buffered media data being a most recent portion of media data received at the first user device, the most recent portion inclusive of the media data received from a present time to a prior time that is fixed relative to the present time; responsive to a search operation invocation at the present time, sending the buffered media data to a search processing system that is remote from the first user device; and receiving, from the search processing system and in response to the buffered media data, contextual information regarding an entity that the data processing system identified from processing the buffered media data.

公开(公告)号：US20200250227A1

公开(公告)日：2020-08-06

申请号：US16731786

申请日：2019-12-31

Applicant: Google LLC

Inventor： Ibrahim Badr ,  Nils Grimsmo ,  Gokhan H. Bakir ,  Kamil Anikiej ,  Aayush Kumar ,  Viacheslav Kuznetsov

IPC: G06F16/58 ,  G06K9/32 ,  G06F16/9032 ,  G06K9/72

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for contextually disambiguating queries are disclosed. In an aspect, a method includes receiving an image being presented on a display of a computing device and a transcription of an utterance spoken by a user of the computing device, identifying a particular sub-image that is included in the image, and based on performing image recognition on the particular sub-image, determining one or more first labels that indicate a context of the particular sub-image. The method also includes, based on performing text recognition on a portion of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image, based on the transcription, the first labels, and the second labels, generating a search query, and providing, for output, the search query.

公开(公告)号：US20200021740A1

公开(公告)日：2020-01-16

申请号：US16585085

申请日：2019-09-27

Applicant: Google LLC

Inventor： Ibrahim Badr ,  Gökhan Bakir ,  Daniel Kunkle ,  Kavin Karthik Ilangovan ,  Denis Burakov

IPC: H04N5/232 ,  G06K9/32 ,  G06K9/00 ,  G06F16/583 ,  H04N9/82 ,  H04N5/77 ,  G06F16/58 ,  H04N1/00 ,  H04N1/32 ,  H04N1/21

Abstract: The present disclosure relates to user-selected metadata related to images captured by a camera of a client device. User-selected metadata may include contextual information and/or information provided by a user when the images are captured. In various implementations, a free form input may be received at a first client device of one or more client devices operated by a user. A task request may be recognized from the free form input, and it may be determined that the task request includes a request to store metadata related to one or more images captured by a camera of the first client device. The metadata may be selected based on content of the task request. The metadata may then be stored, e.g., in association with one or more images captured by the camera, in computer-readable media. The computer-readable media may be searchable by the metadata.