Abstract:
An efficient streaming method and apparatus for detecting hierarchical heavy hitters from massive data streams is disclosed. In one embodiment, the method enables near real time detection of anomaly behavior in networks.
Abstract:
Disclosed herein are systems, computer-implemented methods, and computer-readable media for sampling network traffic. The method includes receiving a plurality of flow records, calculating a hash for each flow record based on one or more invariant part of a respective flow, generating a quasi-random number from the calculated hash for each respective flow record, and sampling flow records having a quasi-random number below a probability P. Invariant parts of flow records include destination IP address, source IP address, TCP/UDP port numbers, TCP flags, and network protocol. A plurality of routers can uniformly calculate hashes for flow records. Each router in a plurality of routers can generate a same quasi-random number for each respective flow record and uses different values for probability P. The probability P can depend on a flow size. The method can divide the quasi-random number by a maximum possible hash value.
Abstract:
The invention relates to streaming algorithms useful for obtaining summaries over unaggregated packet streams and for providing unbiased estimators for characteristics, such as, the amount of traffic that belongs to a specified subpopulation of flows. Packets are sampled from a packet stream and aggregated into flows and counted by implementation of Adaptive Sample-and-Hold (ASH) or Adaptive NetFlow (ANF), adjusting the sampling rate based on a quantity of flows to obtain a sketch having a predetermined size, the sampling rate being adjusted in steps; and transferring the count of aggregated packets from SRAM to DRAM and initializing the count in SRAM following adjustment of the sampling rate.
Abstract:
The invention relates to streaming algorithms useful for obtaining summaries over unaggregated packet streams and for providing unbiased estimators for characteristics, such as, the amount of traffic that belongs to a specified subpopulation of flows. Packets are sampled from a packet stream and aggregated into flows and counted by implementation of: (a) Adaptive Sampled NetFlow (ANF), and adjusted weight (AANF) of a flow (f) is calculated as follows: AANF(f)=i(f)/p′; i(f) being the number of packets counted for a flow f, and p′ being the sampling rate at end of a measurement period; or (b) Adaptive Sample-and-Hold (ASH), and adjusted weight (AASH) of a flow (f) is calculated as follows: AASH(f)=i(f)+(1−p′)/p′; i(f) being the number of packets counted for a flow f, and p′ being the sampling rate at end of a measurement period.
Abstract translation:本发明涉及用于在未分组的分组流上获得摘要的用于提供用于特征的无偏估计器的流式传输算法,例如属于指定的流量子群的业务量。 分组从分组流中采样并聚合成流,并通过实现计算:(a)自适应采样NetFlow(ANF)和流(f)的调整权重(AANF)计算如下:AANF(f)= i (f)/ p'; i(f)是流f计数的分组数,p'是测量周期结束时的采样率; 或(b)自适应采样保持(ASH)和流(f)的调整权重(AASH)如下计算:AASH(f)= i(f)+(1-p')/ p' ; i(f)是流f计数的分组数,p'是测量周期结束时的采样率。
Abstract:
An apparatus for optimizing a filter based on detected attacks on a data network includes an estimation means and an optimization means. The estimation means operates when a detector detects an attack and the detector transmits an inaccurate attack severity. The estimation means determines an accurate attack severity. The optimization means adjusts a parameter and the parameter is an input to a filter.
Abstract:
The present invention develops an efficient streaming method for detecting multidimensional hierarchical heavy hitters from massive data streams and enables near real time detection of anomaly behavior in networks.
Abstract:
Two regularized estimators that avoid the pathologies associated with variance estimation are disclosed. The regularized variance estimator adds a contribution to estimated variance representing the likely error, and hence ameliorates the pathologies of estimating small variances while at the same time allowing more reliable estimates to be balanced in the convex combination estimator. The bounded variance estimator employs an upper bound to the variance which avoids estimation pathologies when sampling probabilities are very small.
Abstract:
An apparatus for optimizing a filter based on detected attacks on a data network includes an estimation means and an optimization means. The estimation means operates when a detector detects an attack and the detector transmits an inaccurate attack severity. The estimation means determines an accurate attack severity. The optimization means adjusts a parameter and the parameter is an input to a filter.