公开(公告)号：US20240362279A1

公开(公告)日：2024-10-31

申请号：US18306638

申请日：2023-04-25

Applicant: Google LLC

Inventor： Harshit Kharbanda ,  Belinda Luna Zeng ,  Viviana Caso Corella ,  Christopher James Kelley ,  Jessica Lee ,  Pendar Yousefi ,  Dounia Berrada ,  Sundeep Vaddadi ,  Kai Yu ,  Balint Miklos ,  Severin Heiniger ,  Louis Wang

IPC: G06F16/9532 ,  G06F16/538 ,  G06F40/40

CPC classification number: G06F16/9532 ,  G06F16/538 ,  G06F40/40

Abstract: A multimodal search system is described. The system can receive image data captured by a camera of a user device. Additionally, the system can receive audio data associated with the image data. The audio data can be captured by a microphone of the user device. Moreover, the system can process the image data to generate visual features. Furthermore, the system can process the audio data to generate a plurality of words. The system can generate a plurality of search terms based on the plurality of words and the visual features. Subsequently, the system can determine one or more search results associated with the plurality of search terms and provide the one or more search results as an output.

公开(公告)号：US20240370487A1

公开(公告)日：2024-11-07

申请号：US18253859

申请日：2022-11-04

Applicant: Google LLC

Inventor： Severin Heiniger ,  Balint Miklos ,  Yun-Hsuan Sung ,  Zhen Li ,  Yinfei Yang ,  Chao Jia

IPC: G06F16/538 ,  G06F16/55 ,  G06N3/084

Abstract: Systems and methods of the present disclosure are directed to computer-implemented method for machine-learned multimodal search refinement. The method includes obtaining a query image embedding for a query image and a textual query refinement associated with the query image. The method includes processing the query image embedding and the textual query refinement with a machine-learned query refinement model to obtain a refined query image embedding that incorporates the textual query refinement. The method includes evaluating a loss function that evaluates a distance between the refined query image embedding and an embedding for a ground truth image within an image embedding space. The method includes modifying value(s) of parameter(s) of the machine-learned query refinement model based on the loss function.

公开(公告)号：USD1042529S1

公开(公告)日：2024-09-17

申请号：US29869040

申请日：2022-12-20

Applicant: Google LLC

Designer： Jessica Lee ,  Bálint Miklos ,  Harshit Kharbanda ,  Severin Heiniger

Abstract: FIG. 1 is a front view of a display screen or portion thereof with a transitional graphical user interface showing a first image of the claimed design; and,
FIG. 2 is a second image thereof.
The outermost evenly spaced broken lines show an electronic device that forms no part of the claimed design. The dot-dash broken lines show a display screen or portion thereof and form no part of the claimed design. The remaining broken lines and all lined-through text, show portions of the transitional graphical user interface, and form no part of the claimed design.
The appearance of the transitional image sequentially transitions between the views of FIGS. 1-2. The process or period in which an image transitions to another forms no part of the claimed design.

公开(公告)号：USD1042528S1

公开(公告)日：2024-09-17

申请号：US29869039

申请日：2022-12-20

Applicant: Google LLC

Designer： Jessica Lee ,  Bálint Miklos ,  Harshit Kharbanda ,  Severin Heiniger

Abstract: FIG. 1 is a front view of a display screen or portion thereof with a transitional graphical user interface showing a first image of the claimed design; and,
FIG. 2 is a second image thereof.
The outermost evenly spaced broken lines show an electronic device that forms no part of the claimed design. The dot-dash broken lines show a display screen or portion thereof and form no part of the claimed design. The remaining broken lines and all lined-through text, show portions of the transitional graphical user interface, and form no part of the claimed design.
The appearance of the transitional image sequentially transitions between the views of FIGS. 1-2. The process or period in which an image transitions to another forms no part of the claimed design.

公开(公告)号：US20240028638A1

公开(公告)日：2024-01-25

申请号：US17871531

申请日：2022-07-22

Applicant: Google LLC

Inventor： Balint Miklos ,  Rajan Sharad Patel ,  Severin Heiniger

IPC: G06F16/532 ,  G06F9/451 ,  G06F16/538

CPC classification number: G06F16/532 ,  G06F9/453 ,  G06F16/538

Abstract: Systems and methods of the present disclosure are directed to a computer-implemented method for multimodal search refinement. The method includes obtaining a visual search query from a user comprising one or more query images. The method includes providing a search interface for display to the user, the search interface comprising one or more result images responsive to the one or more query images and an interface element indicative of a request to the user to refine the visual search query. The method includes obtaining, from the user, textual data comprising a refinement to the visual search query. The method includes appending, by the computing system, the textual data to the visual search query to obtain a multimodal search query.