公开(公告)号：US10142412B2

公开(公告)日：2018-11-27

申请号：US15913079

申请日：2018-03-06

Applicant: Splunk Inc.

Inventor： Sourav Pal ,  Christopher Madden Pride

IPC: G06F15/167 ,  H04L29/08 ,  H04L12/26 ,  G06F17/30

Abstract: Multi-thread processing of search responses is disclosed. An example method may include transmitting, by a computer system, a search request to a plurality of search peers of a data aggregation and analysis system; receiving a plurality of data packets from the plurality of search peers; parsing, by a first processing thread of the computer system, one or more data packets of the plurality of data packets, to produce a partial response to the search request; and processing, by a second processing thread of the computer system, the partial response to produce a memory data structure representing an aggregated response to the search request.

公开(公告)号：US20180089324A1

公开(公告)日：2018-03-29

申请号：US15665339

申请日：2017-07-31

Applicant: Splunk Inc.

Inventor： Sourav Pal ,  Arindam Bhattacharjee ,  Alexander Douglas James

IPC: G06F17/30 ,  G06F9/54

CPC classification number: G06F16/9535 ,  G06F9/5011 ,  G06F9/546 ,  G06F16/2471 ,  G06F16/90335

Abstract: Systems and methods are disclosed for utilizing an ingested data buffer operating according to a publish-subscribe messaging model as an intake mechanism for a query system. Data from various sources can be placed into the data buffer according to different topics. Indexers can subscribe to these topics in order to ingest the data into the system for long-term storage and later search. In addition, worker nodes may directly subscribe to the topics to enable continuous or streaming searching of the data, without delays that may be caused by ingestion of the data at an indexer. When a request for a streaming search is received, a query coordinator can determine a number of message queues on the data buffer that contain potentially relevant messages. The query coordinator can then dynamically allocate partitions operating on worker nodes to retrieve and intake messages from the message queues into a phased search process.

公开(公告)号：US20180089269A1

公开(公告)日：2018-03-29

申请号：US15665148

申请日：2017-07-31

Applicant: Splunk Inc.

Inventor： Sourav Pal ,  Arindam Bhattacharjee ,  Christopher Pride

IPC: G06F17/30

CPC classification number: G06F16/24542 ,  G06F16/24554 ,  G06F16/258

Abstract: Systems and methods are disclosed for processing queries against one or more dataset sources. The system tracks query resource data and resource utilization data. The query-resource usage data can indicate resources used to execute queries. The node resource utilization data can indicate current utilization of nodes in the system. Upon receipt of a query that identifies a set of data to be processed and a manner of processing the set of data, the system can use the query-resource usage data and the resource utilization data to define a query processing scheme. The query can then be executed using the query processing scheme. In some cases, the query coordinator can dynamically allocate partitions operating on worker nodes to execute the query.

公开(公告)号：US20180089262A1

公开(公告)日：2018-03-29

申请号：US15665302

申请日：2017-07-31

Applicant: Splunk Inc.

Inventor： Arindam Bhattacharjee ,  Sourav Pal ,  Ramkumar Chandrasekharan

IPC: G06F17/30

CPC classification number: G06F16/24532 ,  G06F16/24535 ,  G06F16/24554 ,  G06F16/2465 ,  G06F16/3334 ,  G06F16/3349

Abstract: Systems and methods are disclosed for processing queries against a common storage utilizing dynamically allocated partitions operating on one or more worker nodes. The common storage can include one or more data stores, which collectively contain a data set divided across multiple buckets of data. To query the common storage, a query coordinator can retrieve metadata regarding the multiple buckets, in order to determine a subset of buckets that are potentially relevant to a query. The query coordinator can then dynamically allocate partitions operating on worker nodes to retrieve and intake individual buckets of the subset into a phased search process. The dynamic allocation can be selected to maximize parallelization of the buckets across partitions, thus increasing a speed at which the common storage can be searched.

公开(公告)号：US20180089259A1

公开(公告)日：2018-03-29

申请号：US15665248

申请日：2017-07-31

Applicant: Splunk Inc.

Inventor： Alexander Douglas James ,  Sourav Pal ,  Arindam Bhattacharjee ,  Christopher Pride

IPC: G06F17/30

CPC classification number: G06F16/2425 ,  G06F16/2282

Abstract: Systems and methods are disclosed for processing queries against an external data source utilizing dynamically allocated partitions operating on one or more worker nodes. The external data source can include data that has not been processed by the system. To query the external data source, a query coordinator can generate a subquery for the external data source based on determined functionality of the data source. The subquery can identify data in the external data source for processing and a manner for processing the data. In addition, the query coordinator can dynamically allocate partitions operating on worker nodes to retrieve and intake results of the subquery. In some cases, number of partitions allocated can be based on a number of partitions supported by the external data source.

公开(公告)号：US09509765B2

公开(公告)日：2016-11-29

申请号：US14448995

申请日：2014-07-31

Applicant: Splunk Inc.

Inventor： Sourav Pal ,  Christopher Madden Pride

IPC: G06F15/167 ,  H04L29/08 ,  G06F17/30

CPC classification number: H04L67/1087 ,  G06F15/167 ,  G06F17/30864 ,  H04L43/106 ,  H04L43/16 ,  H04L67/02 ,  H04L67/1004

Abstract: Systems and methods for asynchronous processing of messages that are received from multiple servers. An example method may comprise: receiving, by a first processing thread, in a non-blocking mode, a plurality of sub-application layer protocol packets from a plurality of servers; processing one or more sub-application layer protocol packets received from a first server of the plurality of servers, to produce a first application layer message; writing the first application layer message to a message queue; processing one or more sub-application layer protocol packets received from a second server of the plurality of servers, to produce a second application layer message; writing the second application layer message to the message queue; and reading, by two or more processing threads of a processing thread pool, two or more application layer messages including the first application layer message and the second application layer message from the message queue, to produce two or more memory data structures based on the read application layer messages.

Abstract translation: 用于异步处理从多个服务器接收的消息的系统和方法。 示例性方法可以包括：由第一处理线程以非阻塞模式从多个服务器接收多个子应用层协议分组; 处理从所述多个服务器的第一服务器接收的一个或多个子应用层协议分组，以产生第一应用层消息; 将第一应用层消息写入消息队列; 处理从所述多个服务器的第二服务器接收的一个或多个子应用层协议分组，以产生第二应用层消息; 将第二应用层消息写入消息队列; 并且通过处理线程池的两个或更多个处理线程从消息队列读取包括第一应用层消息和第二应用层消息的两个或多个应用层消息，以基于读取生成两个或更多个存储器数据结构 应用层消息。

公开(公告)号：US12072891B1

公开(公告)日：2024-08-27

申请号：US18180728

申请日：2023-03-08

Applicant: Splunk Inc.

Inventor： Sourav Pal ,  Ashish Mathew ,  Xiaowei Wang ,  Christopher Pride

IPC: G06F16/2455 ,  G06F16/22 ,  G06F16/2453 ,  G06F16/2458 ,  G06F16/248 ,  G06F16/951

CPC classification number: G06F16/24564 ,  G06F16/22 ,  G06F16/24532 ,  G06F16/2471 ,  G06F16/248 ,  G06F16/951

Abstract: The disclosed embodiments include a method performed by a data intake and query system. The method includes receiving a search query by a search head, defining a search process for applying the search query to indexers, delegating a first portion of the search process to indexers and a second portion of the search process to intermediary node(s) communicatively coupled to the search head and the indexers. The first portion can define a search scope for obtaining partial search results of the indexers and the second portion can define operations for combining the partial search results by the intermediary node(s) to produce a combination of the partial search results. The search head then receives the combination of the partial search results, and outputs final search results for the search query, where the final search results are based on the combination of the partial search results.

公开(公告)号：US12013895B2

公开(公告)日：2024-06-18

申请号：US18328607

申请日：2023-06-02

Applicant: Splunk Inc.

Inventor： Alexandros Batsakis ,  Ashish Mathew ,  Christopher Madden Pride ,  Bharath Kishore Reddy Aleti ,  Sourav Pal ,  Arindam Bhattacharjee ,  James Monschke

IPC: G06F16/23 ,  G06F3/06 ,  G06F16/27 ,  G06F16/901 ,  G06F16/903

CPC classification number: G06F16/901 ,  G06F3/0604 ,  G06F3/0644 ,  G06F3/065 ,  G06F3/0652 ,  G06F3/0653 ,  G06F3/0656 ,  G06F3/067 ,  G06F16/23 ,  G06F16/27 ,  G06F16/903

Abstract: Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.

公开(公告)号：US11989194B2

公开(公告)日：2024-05-21

申请号：US16657867

申请日：2019-10-18

Applicant: Splunk Inc.

Inventor： Arindam Bhattacharjee ,  Sourav Pal ,  Srinivas Bobba

IPC: G06F16/2458 ,  G06F16/27

CPC classification number: G06F16/2471 ,  G06F16/278

Abstract: Systems and methods are described for distributed processing a query in a first query language utilizing a query execution engine intended for single-device execution. While distributed processing provides numerous benefits over single-device processing, distributed query execution engines can be significantly more difficult to develop that single-device engines. Embodiments of this disclosure enable the use of a single-device engine to support distributed processing, by dividing a query into multiple stages, each of which can be executed by multiple, concurrent executions of a single-device engine. Between stages, data can be shuffled between executions of the engine, such that individual executions of the engine are provided with a complete set of records needed to implement an individual stage. Because single-device engines can be significantly less difficult to develop, use of the techniques described herein can enable a distributed system to rapidly support multiple query languages.

公开(公告)号：US11695830B1

公开(公告)日：2023-07-04

申请号：US17722005

申请日：2022-04-15

Applicant: Splunk Inc.

Inventor： Sourav Pal ,  Christopher Madden Pride

IPC: G06F15/167 ,  H04L67/1087 ,  H04L67/1004 ,  H04L67/02 ,  H04L43/106 ,  H04L43/16 ,  G06F16/951

CPC classification number: H04L67/1087 ,  G06F15/167 ,  G06F16/951 ,  H04L43/106 ,  H04L43/16 ,  H04L67/02 ,  H04L67/1004

Abstract: Multi-threaded processing of search responses returned by search peers is disclosed. An example method may include transmitting, by a computer system, a search request to a plurality of search peers of a data aggregation and analysis system; receiving, by a first processing thread, a plurality of data packets from the plurality of search peers; parsing, by a second processing thread, one or more data packets of the plurality of data packets to produce a first partial response to the search request; parsing, by a third processing thread, the one or more data packets to produce a second partial response to the search request; and generating, based on the first partial response and the second partial response, an aggregated response to the search request.