公开(公告)号：US11775254B2

公开(公告)日：2023-10-03

申请号：US17251468

申请日：2020-01-31

Applicant: Google LLC

Inventor： Joseph Lange ,  Asier Aguirre ,  Olivier Siegenthaler ,  Michal Pryt

IPC: G06F3/16 ,  G06T7/70 ,  G06V20/00 ,  G10L15/26

CPC classification number: G06F3/167 ,  G06T7/70 ,  G06V20/00 ,  G10L15/26 ,  G06T2200/24

Abstract: Implementations are described herein for analyzing existing graphical user interfaces (“GUIs”) to facilitate automatic interaction with those GUIs, e.g., by automated assistants or via other user interfaces, with minimal effort from the hosts of those GUIs. For example, in various implementations, a user intent to interact with a particular GUI may be determined based at least in part on a free-form natural language input. Based on the user intent, a target visual cue to be located in the GUI may be identified, and object recognition processing may be performed on a screenshot of the GUI to determine a location of a detected instance of the target visual cue in the screenshot. Based on the location of the detected instance of the target visual cue, an interactive element of the GUI may be identified and automatically populate with data determined from the user intent.

公开(公告)号：US20250060934A1

公开(公告)日：2025-02-20

申请号：US18934902

申请日：2024-11-01

Applicant: GOOGLE LLC

Inventor： Joseph Lange ,  Asier Aguirre ,  Olivier Siegenthaler ,  Michal Pryt

IPC: G06F3/16 ,  G06T7/70 ,  G06V20/00 ,  G10L15/26

Abstract: Implementations are described herein for analyzing existing graphical user interfaces (“GUIs”) to facilitate automatic interaction with those GUIs, e.g., by automated assistants or via other user interfaces, with minimal effort from the hosts of those GUIs. For example, in various implementations, a user intent to interact with a particular GUI may be determined based at least in part on a free-form natural language input. Based on the user intent, a target visual cue to be located in the GUI may be identified, and object recognition processing may be performed on a screenshot of the GUI to determine a location of a detected instance of the target visual cue in the screenshot. Based on the location of the detected instance of the target visual cue, an interactive element of the GUI may be identified and automatically populate with data determined from the user intent.

公开(公告)号：US12147732B2

公开(公告)日：2024-11-19

申请号：US18234760

申请日：2023-08-16

Applicant: GOOGLE LLC

Inventor： Joseph Lange ,  Asier Aguirre ,  Olivier Siegenthaler ,  Michal Pryt

IPC: G06F3/16 ,  G06T7/70 ,  G06V20/00 ,  G10L15/26

Abstract: Implementations are described herein for analyzing existing graphical user interfaces (“GUIs”) to facilitate automatic interaction with those GUIs, e.g., by automated assistants or via other user interfaces, with minimal effort from the hosts of those GUIs. For example, in various implementations, a user intent to interact with a particular GUI may be determined based at least in part on a free-form natural language input. Based on the user intent, a target visual cue to be located in the GUI may be identified, and object recognition processing may be performed on a screenshot of the GUI to determine a location of a detected instance of the target visual cue in the screenshot. Based on the location of the detected instance of the target visual cue, an interactive element of the GUI may be identified and automatically populate with data determined from the user intent.

公开(公告)号：US20230393810A1

公开(公告)日：2023-12-07

申请号：US18234760

申请日：2023-08-16

Applicant: GOOGLE LLC

Inventor： Joseph Lange ,  Asier Aguirre ,  Olivier Siegenthaler ,  Michal Pryt

IPC: G06F3/16 ,  G06T7/70 ,  G06V20/00 ,  G10L15/26

CPC classification number: G06F3/167 ,  G06T7/70 ,  G06V20/00 ,  G10L15/26 ,  G06T2200/24

Abstract: Implementations are described herein for analyzing existing graphical user interfaces (“GUIs”) to facilitate automatic interaction with those GUIs, e.g., by automated assistants or via other user interfaces, with minimal effort from the hosts of those GUIs. For example, in various implementations, a user intent to interact with a particular GUI may be determined based at least in part on a free-form natural language input. Based on the user intent, a target visual cue to be located in the GUI may be identified, and object recognition processing may be performed on a screenshot of the GUI to determine a location of a detected instance of the target visual cue in the screenshot. Based on the location of the detected instance of the target visual cue, an interactive element of the GUI may be identified and automatically populate with data determined from the user intent.