公开(公告)号：US20140330822A1

公开(公告)日：2014-11-06

申请号：US14336464

申请日：2014-07-21

Applicant: GOOGLE INC.

Inventor： Ameesh Makadia ,  Jason E. Weston

IPC: G06F17/30

CPC classification number: G06F16/435 ,  G06F16/433 ,  G06F16/434 ,  G06F16/438

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing joint image-audio queries. In one aspect, a method includes receiving, from a client device, a joint image-audio query including query image data and query audio data. Query image feature data is determined from the query image data. Query audio feature data is determined from the audio data. The query image feature data and the query audio feature data are provided to a joint image-audio relevance model trained to generate relevance scores for a plurality of resources, each resource including resource image data defining a resource image for the resource and text data defining resource text for the resource. Each relevance score is a measure of the relevance of corresponding resource to the joint image-audio query. Data defining search results indicating the order of the resources is provided to the client device.

Abstract translation: 方法，系统和装置，包括在计算机存储介质上编码的计算机程序，用于处理联合图像 - 音频查询。 一方面，一种方法包括从客户端设备接收包括查询图像数据和查询音频数据的联合图像 - 音频查询。 从查询图像数据确定查询图像特征数据。 从音频数据确定查询音频特征数据。 将查询图像特征数据和查询音频特征数据提供给被训练为生成多个资源的相关性分数的联合图像 - 音频相关性模型，每个资源包括定义资源的资源图像的资源图像数据和定义资源的文本数据 资源的文本。 每个相关性分数是对应资源与联合图像 - 音频查询的相关性的度量。 将指示资源顺序的搜索结果的数据提供给客户端设备。