公开(公告)号：US20130246457A1

公开(公告)日：2013-09-19

申请号：US13826502

申请日：2013-03-14

申请人： Mihailo M. Stojancic ,  Sunil Suresh Kulkarni ,  Shashank Merchant ,  Jose Pio Pereira ,  Oleksiy Bolgarov

发明人： Mihailo M. Stojancic ,  Sunil Suresh Kulkarni ,  Shashank Merchant ,  Jose Pio Pereira ,  Oleksiy Bolgarov

IPC分类号： G06F17/30

CPC分类号： G06F17/30023 ,  G06F17/30026 ,  G06F17/30047

摘要： Techniques for efficient database formation and search in applications embedded in a media device are provided. The search may be performed synchronously with presentation of media programming content on a nearby media presentation device. A mobile media device captures some temporal fragments of the presented audio/video content on its microphone and camera, and then generates query fingerprints for the captured fragment. A local reference database resides on the mobile media device and a master reference database resides on a remote server with a most recent chunk of reference fingerprints transferred dynamically to the local mobile media device. A chunk of the query fingerprints generated locally on the mobile media device are searched on the local reference database for continuous content search and identification. The method presented automatically switches between the local search on the mobile media device and a remote search on an external search server.

摘要翻译： 提供了用于在嵌入在媒体设备中的应用中有效的数据库形成和搜索的技术。 搜索可以与附近的媒体呈现设备上的媒体节目内容的呈现同步地执行。 移动媒体设备捕获其麦克风和相机上呈现的音频/视频内容的一些时间片段，然后为捕获的片段生成查询指纹。 本地参考数据库驻留在移动媒体设备上，主参考数据库驻留在远程服务器上，最近一批参考指纹被动态传输到本地移动媒体设备。 在本地参考数据库上搜索在移动媒体设备上本地生成的查询指纹的块，用于连续内容搜索和识别。 所呈现的方法自动地在移动媒体设备上的本地搜索和外部搜索服务器上的远程搜索之间进行切换。

公开(公告)号：US09367544B2

公开(公告)日：2016-06-14

申请号：US13826502

申请日：2013-03-14

申请人： Mihailo M. Stojancic ,  Sunil Suresh Kulkarni ,  Shashank Merchant ,  Jose Pio Pereira ,  Oleksiy Bolgarov

发明人： Mihailo M. Stojancic ,  Sunil Suresh Kulkarni ,  Shashank Merchant ,  Jose Pio Pereira ,  Oleksiy Bolgarov

IPC分类号： G06F17/30

CPC分类号： G06F17/30023 ,  G06F17/30026 ,  G06F17/30047

摘要： Techniques for efficient database formation and search in applications embedded in a media device are provided. The search may be performed synchronously with presentation of media programming content on a nearby media presentation device. A mobile media device captures some temporal fragments of the presented audio/video content on its microphone and camera, and then generates query fingerprints for the captured fragment. A local reference database resides on the mobile media device and a master reference database resides on a remote server with a most recent chunk of reference fingerprints transferred dynamically to the local mobile media device. A chunk of the query fingerprints generated locally on the mobile media device are searched on the local reference database for continuous content search and identification. The method presented automatically switches between the local search on the mobile media device and a remote search on an external search server.

摘要翻译： 提供了用于在嵌入在媒体设备中的应用中有效的数据库形成和搜索的技术。 搜索可以与附近的媒体呈现设备上的媒体节目内容的呈现同步地执行。 移动媒体设备捕获其麦克风和相机上呈现的音频/视频内容的一些时间片段，然后为捕获的片段生成查询指纹。 本地参考数据库驻留在移动媒体设备上，主参考数据库驻留在远程服务器上，最近一批参考指纹被动态传输到本地移动媒体设备。 在本地参考数据库上搜索在移动媒体设备上本地生成的查询指纹的块，用于连续内容搜索和识别。 所呈现的方法自动地在移动媒体设备上的本地搜索和外部搜索服务器上的远程搜索之间进行切换。

公开(公告)号：US09313359B1

公开(公告)日：2016-04-12

申请号：US13590701

申请日：2012-08-21

申请人： Mihailo M. Stojancic ,  Jose Pio Pereira ,  Peter Wendt ,  Shashank Merchant ,  Sunil Suresh Kulkarni

发明人： Mihailo M. Stojancic ,  Jose Pio Pereira ,  Peter Wendt ,  Shashank Merchant ,  Sunil Suresh Kulkarni

IPC分类号： H04N1/32 ,  H04N5/44 ,  H04N21/439

CPC分类号： H04N21/4394 ,  G10H1/0008 ,  G10H2210/041 ,  G10H2210/051 ,  G10H2240/141 ,  G10H2250/235 ,  H04N1/32 ,  H04N21/41407 ,  H04N21/42203 ,  H04N21/4223 ,  H04N21/4332 ,  H04N21/439 ,  H04N21/8352 ,  H04N21/84 ,  H04N21/8456

摘要： A mobile device responds in real time to media content presented on a media device, such as a television. The mobile device captures temporal fragments of audio-video content on its microphone, camera, or both and generates corresponding audio-video query fingerprints. The query fingerprints are transmitted to a search server located remotely or used with a search function on the mobile device for content search and identification. Audio features are extracted and audio signal global onset detection is used for input audio frame alignment. Additional audio feature signatures are generated from local audio frame onsets, audio frame frequency domain entropy, and maximum change in the spectral coefficients. Video frames are analyzed to find a television screen in the frames, and a detected active television quadrilateral is used to generate video fingerprints to be combined with audio fingerprints for more reliable content identification.

摘要翻译： 移动设备实时响应媒体设备（如电视机）上呈现的媒体内容。 移动设备捕获其麦克风，相机或两者上的音频 - 视频内容的时间片段，并生成相应的音视频查询指纹。 查询指纹被发送到位于远程的搜索服务器或与移动设备上的搜索功能一起使用，用于内容搜索和识别。 提取音频特征，音频信号全局起始检测用于输入音频帧对齐。 额外的音频特征签名由本地音频帧发送，音频帧频域熵和频谱系数的最大变化产生。 视频帧被分析以在帧中找到电视屏幕，并且使用检测到的活动电视四边形来生成视频指纹以与音频指纹组合以用于更可靠的内容识别。

公开(公告)号：US20150254344A1

公开(公告)日：2015-09-10

申请号：US14718218

申请日：2015-05-21

申请人： Sunil Suresh Kulkarni ,  Jose Pio Pereira ,  Pradipkumar Dineshbhai Gajjar ,  Shashank Merchant ,  Prashant Ramanathan ,  Mihailo M. Stojancic

发明人： Sunil Suresh Kulkarni ,  Jose Pio Pereira ,  Pradipkumar Dineshbhai Gajjar ,  Shashank Merchant ,  Prashant Ramanathan ,  Mihailo M. Stojancic

IPC分类号： G06F17/30

CPC分类号： G06F17/30799 ,  G06F17/30023 ,  G06F17/3033 ,  G06F17/30424 ,  G06F17/30864

摘要： An architecture for a multimedia search system is described. To perform similarity matching of multimedia query frames against reference content, reference database comprising of a cluster index using cluster keys to perform similarity matching and a multimedia index to perform sequence matching is built. Methods to update and maintain the reference database that enables addition and removal of the multimedia contents, including portions of multimedia content, from the reference database in a running system are described. Hierarchical multi-level partitioning methods to organize the reference database are presented. Smart partitioning of the reference multimedia content according to the nature of the multimedia content, and according to the popularity among the social media, that supports scalable fast multimedia identification is also presented. A caching mechanism for multimedia search queries in a centralized or in a decentralized distributed system and a client based local multimedia search system enabling multimedia tracking are described.

摘要翻译： 描述了用于多媒体搜索系统的架构。 为了执行多媒体查询帧与参考内容的相似性匹配，构建了使用集群密钥进行相似性匹配的集群索引的参考数据库和执行序列匹配的多媒体索引。 描述了在运行系统中更新和维护参考数据库的方法，该参考数据库能够从参考数据库添加和删除包括多媒体内容的多媒体内容的多媒体内容。 提出了组织参考数据库的分层多级分区方法。 根据多媒体内容的性质，根据社交媒体的流行度，对参考多媒体内容进行智能划分，支持可扩展快速多媒体识别。 描述了一种用于集中式或分散式分布式系统中的多媒体搜索查询的缓存机制以及启用多媒体跟踪的基于客户端的本地多媒体搜索系统。

公开(公告)号：US09058355B1

公开(公告)日：2015-06-16

申请号：US14151335

申请日：2014-01-09

申请人： Sunil Suresh Kulkarni ,  Jose Pio Pereira ,  Pradipkumar Dineshbhai Gajjar ,  Shashank Merchant ,  Prashant Ramanathan ,  Mihailo M. Stojancic

发明人： Sunil Suresh Kulkarni ,  Jose Pio Pereira ,  Pradipkumar Dineshbhai Gajjar ,  Shashank Merchant ,  Prashant Ramanathan ,  Mihailo M. Stojancic

IPC分类号： G06F17/30

CPC分类号： G06F17/30799 ,  G06F17/30023 ,  G06F17/3033 ,  G06F17/30424 ,  G06F17/30864

摘要： An architecture for a multimedia search system is described. To perform similarity matching of multimedia query frames against reference content, reference database comprising of a cluster index using cluster keys to perform similarity matching and a multimedia index to perform sequence matching is built. Methods to update and maintain the reference database that enables addition and removal of the multimedia contents, including portions of multimedia content, from the reference database in a running system are described. Hierarchical multi-level partitioning methods to organize the reference database are presented. Smart partitioning of the reference multimedia content according to the nature of the multimedia content, and according to the popularity among the social media, that supports scalable fast multimedia identification is also presented. A caching mechanism for multimedia search queries in a centralized or in a decentralized distributed system and a client based local multimedia search system enabling multimedia tracking are described.

摘要翻译： 描述了用于多媒体搜索系统的架构。 为了执行多媒体查询帧与参考内容的相似性匹配，构建了使用集群密钥进行相似性匹配的集群索引的参考数据库和执行序列匹配的多媒体索引。 描述了在运行系统中更新和维护参考数据库的方法，该参考数据库能够从参考数据库添加和删除包括多媒体内容的多媒体内容的多媒体内容。 提出了组织参考数据库的分层多级分区方法。 根据多媒体内容的性质，根据社交媒体的流行度，对参考多媒体内容进行智能划分，支持可扩展快速多媒体识别。 描述了一种用于集中式或分散式分布式系统中的多媒体搜索查询的缓存机制以及启用多媒体跟踪的基于客户端的本地多媒体搜索系统。

公开(公告)号：US08965863B1

公开(公告)日：2015-02-24

申请号：US14151294

申请日：2014-01-09

申请人： Sunil Suresh Kulkarni ,  Jose Pio Pereira ,  Pradipkumar Dineshbhai Gajjar ,  Shashank Merchant ,  Prashant Ramanathan ,  Mihailo M. Stojancic

发明人： Sunil Suresh Kulkarni ,  Jose Pio Pereira ,  Pradipkumar Dineshbhai Gajjar ,  Shashank Merchant ,  Prashant Ramanathan ,  Mihailo M. Stojancic

IPC分类号： G06F17/00 ,  G06F17/30

CPC分类号： G06F17/30799 ,  G06F17/30023 ,  G06F17/3033 ,  G06F17/30424 ,  G06F17/30864

摘要： An architecture for a multimedia search system is described. To perform similarity matching of multimedia query frames against reference content, reference database comprising of a cluster index using cluster keys to perform similarity matching and a multimedia index to perform sequence matching is built. Methods to update and maintain the reference database that enables addition and removal of the multimedia contents, including portions of multimedia content, from the reference database in a running system are described. Hierarchical multi-level partitioning methods to organize the reference database are presented. Smart partitioning of the reference multimedia content according to the nature of the multimedia content, and according to the popularity among the social media, that supports scalable fast multimedia identification is also presented. A caching mechanism for multimedia search queries in a centralized or in a decentralized distributed system and a client based local multimedia search system enabling multimedia tracking are described.

公开(公告)号：US09510044B1

公开(公告)日：2016-11-29

申请号：US13327359

申请日：2011-12-15

申请人： Jose Pio Pereira ,  Sunil Suresh Kulkarni ,  Oleksiy Bolgarov ,  Prashant Ramanathan ,  Shashank Merchant ,  Mihailo Stojancic

发明人： Jose Pio Pereira ,  Sunil Suresh Kulkarni ,  Oleksiy Bolgarov ,  Prashant Ramanathan ,  Shashank Merchant ,  Mihailo Stojancic

IPC分类号： H04H60/32 ,  H04N21/44 ,  H04N21/8352 ,  H04H60/37 ,  H04H60/56 ,  H04H60/59

CPC分类号： H04N21/44008 ,  G06F21/10 ,  G06F2221/0704 ,  G06K9/00758 ,  H04H20/106 ,  H04H60/37 ,  H04H60/56 ,  H04H60/59 ,  H04H2201/90 ,  H04N21/44204 ,  H04N21/8352 ,  H04N21/8456

摘要： Content segmentation, categorization and identification methods are described. Content tracking approaches are illustrated that are suitable for large scale deployment. Time-aligned applications such as multi-language selection, customized advertisements, second screen services and content monitoring applications can be economically deployed at large scales. A client performs fingerprinting, scene change detection, audio turn detection, and logo detection on incoming video and gathers database search results, logos and text to identify and segment video streams into content, promos, and commercials. A learning engine is configured to learn rules for optimal identification and segmentation at each client for each channel and program. Content sensed at the client site is tracked with reduced computation and applications are executed with timing precision. A user interface for time-aligned publishing of content and subsequent usage and interaction on one or more displays is also described.

摘要翻译： 描述内容分割，分类和识别方法。 说明适用于大规模部署的内容跟踪方法。 诸如多语言选择，定制广告，第二屏幕服务和内容监控应用之类的时间对应的应用可以大规模经济地部署。 客户端对进入的视频执行指纹识别，场景变化检测，音频转向检测和标识检测，并收集数据库搜索结果，徽标和文本，以将视频流识别和分割为内容，宣传和广告。 学习引擎配置为每个通道和程序的每个客户端学习最佳识别和分段的规则。 通过减少的计算跟踪在客户端站点处感测的内容，并以时序精度执行应用程序。 还描述了用于在一个或多个显示器上时间对齐地发布内容和随后的使用和交互的用户界面。

公开(公告)号：US20150112988A1

公开(公告)日：2015-04-23

申请号：US14589366

申请日：2015-01-05

申请人： Jose Pio Pereira ,  Shashank Merchant ,  Prashant Ramanathan ,  Sunil Suresh Kulkarni ,  Mihailo Stojancic

发明人： Jose Pio Pereira ,  Shashank Merchant ,  Prashant Ramanathan ,  Sunil Suresh Kulkarni ,  Mihailo Stojancic

IPC分类号： G06F17/30

CPC分类号： G06F17/30038 ,  G06F17/3002 ,  G06F17/30029 ,  G06F17/30377 ,  G06F17/3056

摘要： An efficient large scale search system for video and multi-media content using a distributed database and search, and tiered search servers is described. Selected content is stored at the distributed local database and tier1 search server(s). Content matching frequent queries, and frequent unidentified queries are cached at various levels in the search system. Content is classified using feature descriptors and geographical aspects, at feature level and in time segments. Queries not identified at clients and tier1 search server(s) are queried against tier2 or lower search server(s). Search servers use classification and geographical partitioning to reduce search cost. Methods for content tracking and local content searching are executed on clients. The client performs local search, monitoring and/or tracking of the query content with the reference content and local search with a database of reference fingerprints. This shifts the content search workload from central servers to the distributed monitoring clients.

摘要翻译： 描述了使用分布式数据库和搜索以及分层搜索服务器的视频和多媒体内容的高效大规模搜索系统。 所选内容存储在分布式本地数据库和tier1搜索服务器上。 内容匹配频繁查询和频繁的不明查询在搜索系统的各个级别进行缓存。 内容使用特征描述符和地理方面，在特征级别和时间段中进行分类。 在客户端和tier1搜索服务器上未识别的查询被查询到tier2或更低的搜索服务器。 搜索服务器使用分类和地理分区来降低搜索成本。 在客户端执行内容跟踪和本地内容搜索的方法。 客户端用参考内容和具有参考指纹数据库的本地搜索来执行查询内容的本地搜索，监视和/或跟踪。 这将内容搜索工作负载从中央服务器转移到分布式监控客户端。

公开(公告)号：US08655878B1

公开(公告)日：2014-02-18

申请号：US13102479

申请日：2011-05-06

申请人： Sunil Suresh Kulkarni ,  Jose Pio Pereira ,  Pradipkumar Dineshbhai Gajjar ,  Shashank Merchant ,  Prashant Ramanathan ,  Mihailo Stojancic

发明人： Sunil Suresh Kulkarni ,  Jose Pio Pereira ,  Pradipkumar Dineshbhai Gajjar ,  Shashank Merchant ,  Prashant Ramanathan ,  Mihailo Stojancic

IPC分类号： G06F17/30 ,  G06F7/00

CPC分类号： G06F17/30799 ,  G06F17/30023 ,  G06F17/3033 ,  G06F17/30424 ,  G06F17/30864

摘要： An architecture for a multimedia search system is described. To perform similarity matching of multimedia query frames against reference content, reference database comprising of a cluster index using cluster keys to perform similarity matching and a multimedia index to perform sequence matching is built. Methods to update and maintain the reference database that enables addition and removal of the multimedia contents, including portions of multimedia content, from the reference database in a running system are described. Hierarchical multi-level partitioning methods to organize the reference database are presented. Smart partitioning of the reference multimedia content according to the nature of the multimedia content, and according to the popularity among the social media, that supports scalable fast multimedia identification is also presented. A caching mechanism for multimedia search queries in a centralized or in a decentralized distributed system and a client based local multimedia search system enabling multimedia tracking are described.

摘要翻译： 描述了用于多媒体搜索系统的架构。 为了执行多媒体查询帧与参考内容的相似性匹配，构建了使用集群密钥进行相似性匹配的集群索引的参考数据库和执行序列匹配的多媒体索引。 描述了在运行系统中更新和维护参考数据库的方法，该参考数据库能够从参考数据库添加和删除包括多媒体内容的多媒体内容的多媒体内容。 提出了组织参考数据库的分层多级分区方法。 根据多媒体内容的性质，根据社交媒体的流行度，对参考多媒体内容进行智能划分，支持可扩展快速多媒体识别。 描述了一种用于集中式或分散式分布式系统中的多媒体搜索查询的缓存机制以及启用多媒体跟踪的基于客户端的本地多媒体搜索系统。

公开(公告)号：US08335786B2

公开(公告)日：2012-12-18

申请号：US12788796

申请日：2010-05-27

申请人： Jose Pio Pereira ,  Sunil Suresh Kulkarni ,  Shashank Merchant ,  Prashant Ramanathan ,  Pradipkumar Dineshbhai Gajjar

发明人： Jose Pio Pereira ,  Sunil Suresh Kulkarni ,  Shashank Merchant ,  Prashant Ramanathan ,  Pradipkumar Dineshbhai Gajjar

IPC分类号： G06F17/30

CPC分类号： G06F17/30784 ,  G06F17/30825 ,  G06K9/00758

摘要： A method is presented for large media data base query and media entry identification based on multi-level similarity search and reference-query entry correlation. Media content fingerprinting detects unique features and generates discriminative descriptors and signatures used to form preliminary reference data base. The preliminary reference data base is processed and a subset-set of it is selected to form a final reference data base. To identify a media query a fast similarity search is performed first on the reference database resulting in a preliminary set of likely matching videos. For each preliminary likely matching video a further multi-level correlation is performed which includes iterative refinement, sub-sequence merging, and final result classification.

摘要翻译： 提出了一种基于多级相似搜索和参考查询条目相关性的大型媒体数据库查询和媒体条目标识的方法。 媒体内容指纹识别检测独特的特征，并产生用于形成初步参考数据库的识别描述符和签名。 处理初步参考数据库并选择其初始参考数据库以形成最终参考数据库。 为了识别媒体查询，首先在参考数据库上执行快速相似性搜索，导致可能匹配的视频的初步集合。 对于每个初步可能的匹配视频，执行进一步的多级相关，其包括迭代细化，子序列合并和最终结果分类。