Abstract:
A session table includes one or more records, where each record represents a session. Session record information is stored in various fields, such as key fields, value fields, and timestamp fields. Session information is described as keys and values in order to support query/lookup operations. A session table is associated with a filter, which describes a set of keys that can be used for records in that table. A session table is populated using data contained in security information/events. Rules are created to identify events related to session information, extract the session information, and use the session information to modify a session table. A session table is partitioned so that the number of records in each session table partition is decreased. A session table is processed periodically so that active sessions are moved to the current partition.
Abstract:
A system and method for building merged events from log entries received from multiple devices. Multiple log events generally contribute to a single merged event. In the described embodiment, the mapping module receives log entries associated with specific merged events and maps them to fields in the merged event data structure in accordance with mapping properties. The described embodiments of the invention use regular expressions in the merge properties to describe values that are searched for in the received log entries. A described embodiment of the present invention gives the mapping module access to the event under construction. A new conditional operator, _oneOf, is introduced that selects the first token that is bound to a value out of a list of tokens.
Abstract:
A traditional structured data store is leveraged to provide the benefits of an unstructured full-text search system. A fixed number of "extended" columns is added to the traditional structured data store to form an "enhanced structured data store" (ESDS). The extended columns are independent of any regular columnar interpretation of the data and enable the data that they store to be searched using standard full-text query syntax/techniques that can be executed faster (as opposed to SQL syntax). In other words, the added columns act as a search index. A token is stored in an appropriate extended column based on that token's hash value. The hash value is determined using a hashing scheme, which operates based on the value of the token, rather than the meaning of the token. This enables subsequent searches to be expressed as full-text queries without degrading the ensuing search to a brute force scan.
Abstract:
Patterns can be discovered in security events collected by a network security system (10). In one embodiment, the present invention includes collecting and storing security events from a variety of monitor devices (12).In one embodiment , a subset of the stored security events is provided to a manager (14) as an event stream. In one embodiment, the present invention further includes the manager discovering one or more previously unknown event patterns in the event stream.
Abstract:
An "unstructured event parser" analyzes an event that is in unstructured form and generates an event that is in structured form. A mapping phase determines, for a given event token, possible fields of the structured event schema to which the token could be mapped and the probabilities that the token should be mapped to those fields. Particular tokens are then mapped to particular fields of the structured event schema. By using the Naϊve Bayesian probability model, a "probabilistic mapper" determines, for a particular token and a particular field, the probability that token maps to that field. The probabilistic mapper can also be used in a "regular expression creator" that generates a regex that matches an unstructured event and a "parameter file creator" that helps a user create a parameter file for use with a parameterized normalized event generator to generate a normalized event based on an unstructured event.
Abstract:
Λ unique identifier is assigned to a network node and is used to obtain an "asset model" corresponding to the node and to determine whether the node is a member of a particular category. An asset model is a set of information about a node (e.g., the node's role within the enterprise, software installed on the node, and known vulnerabilities/weaknesses of the node). An identifier lookup module determines a node's identifier based on characteristics of the node (such as [P address., host name, network zone, and/or MAC address), which are used as keys into lookup data structures. A category lookup module determines whether a particular node is a member of (i.e., within) a particular category using a transitive closure to model the categories (properties) that can be attached to an asset model. A transitive closure for a particular asset category is stored as a bitmap, similar to bitmap indexing.
Abstract:
A system for generating a parser and using the parser to parse a target file includes a target file description, an output format description, a Parser generator, a Parser, a target file, and a result object. The target file description and the output format description are included in one or more "properties files", which are text files that include one or more name/value pairs ("properties"). The target file description and the output format description are input into the Parser generator, which outputs the Parser. The target file is input into the Parser, which outputs the result object. The target file description specifies one or more parsers and/or tokenizers that can be used to parse the target file. The parsers and/or tokenizers specified by the target file description are part of the generated Parser. These parsers and/or tokenizers make the Parser more flexible, which enables the Parser to parse semi-structured data.
Abstract:
A system and method for building merged events from log entries received from multiple devices. Multiple log events generally contribute to a single merged event. In the described embodiment, the mapping module (120) receives log entries associated with specific merged events and maps them to fields in the merged event data structure in accordance with mapping properties (122). The described embodiments of the invention use regular expressions in the merge properties (112) to describe values that are searched for in the received log entries. A described embodiment of the present invention gives the mapping module access to the event under construction. A new conditional operator, oneOf, i introduced that selects the first token that is bound to a value out of a list of tokens.
Abstract:
A session table includes one or more records, where each record represents a session. Session record information is stored in various fields, such as key fields, value fields, and timestamp fields. Session information is described as keys and values in order to support query/lookup operations. A session table is associated with a filter, which describes a set of keys that can be used for records in that table. A session table is populated using data contained in security information/events. Rules are created to identify events related to session information, extract the session information, and use the session information to modify a session table. A session table is partitioned so that the number of records in each session table partition is decreased. A session table is processed periodically so that active sessions are moved to the current partition.
Abstract:
A logging system includes an event receiver and a storage manager. The receiver receives log data, processes it, and outputs a data "chunk." The manager receives data chunks and stores them so that they can be queried. The receiver includes buffers that store events and a metadata structure that stores metadata about the contents of the buffers. The metadata includes a unique identifier associated with the receiver, the number of events in the buffers, and, for each "field of interest," a minimum value and a maximum value that reflect the range of values of that field over all of the events in the buffers. A chunk includes the metadata structure and a compressed version of the contents of the buffers. The metadata structure acts as a search index when querying event data. The logging system can be used in conjunction with a security information/event management (SIEM) system.