System and method for graph based monitoring and management of distributed systems

    公开(公告)号:US10353800B2

    公开(公告)日:2019-07-16

    申请号:US15786790

    申请日:2017-10-18

    Abstract: Systems, methods, and computer-readable media are disclosed for graph based monitoring and management of network components of a distributed streaming system. In one aspect, a method includes generating, by a processor, a first metrics and a second metrics based on data collected on a system; generating, by the processor, a topology graph representing data flow within the system; generating, by the processor, at least one first metrics graph corresponding to the first metrics based in part on the topology graph; generating, by the processor, at least one second metrics graph corresponding to the second metrics based in part on the topology graph; identifying, by the processor, a malfunction within the system based on a change in at least one of the first metrics graph and the second metrics graph; and sending, by the processor, a feedback on the malfunction to an operational management component of the system.

    USING PERSISTENT MEMORY TO ENABLE CONSISTENT DATA FOR BATCH PROCESSING AND STREAMING PROCESSING

    公开(公告)号:US20190197146A1

    公开(公告)日:2019-06-27

    申请号:US15850168

    申请日:2017-12-21

    Abstract: Systems, methods, and computer-readable media are provided for consistent data to be used for streaming and batch processing. The system includes one or more devices; a processor coupled to the one or more devices; and a non-volatile memory coupled to the processor and the one or more devices, wherein the non-volatile memory stores instructions that are configured to cause the processor to perform operations including receiving data from the one or more devices; validating the data to yield validated data; storing the validated data in a database on the non-volatile memory, the validated data being used for streaming processing and batch processing; and sending the validated data to a remote disk for batch processing.

    OPTIMIZING PLACEMENT OF VIRTUAL MACHINES
    34.
    发明申请

    公开(公告)号:US20170346759A1

    公开(公告)日:2017-11-30

    申请号:US15682091

    申请日:2017-08-21

    Abstract: Systems and methods are described for allocating resources in a cloud computing environment. The method includes receiving a computing request, the request for use of at least one virtual machine and a portion of memory. In response to the request, a plurality of hosts is identified and a cost function is formulated using at least a portion of those hosts. Based on the cost function, at least one host that is capable of hosting the virtual machine and memory is selected.

    Elastic scale out policy service
    35.
    发明授权
    Elastic scale out policy service 有权
    弹性扩展政策服务

    公开(公告)号:US09560119B2

    公开(公告)日:2017-01-31

    申请号:US14581783

    申请日:2014-12-23

    Abstract: In one embodiment, a scale out policy service for processing a stream of messages includes a distributed stream processing computation system comprising distributed stream processing nodes, a distributed storage system, and a rules engine. A stream processing engine of the distributed stream processing computation system can receive the stream of messages comprising requests and/or events, and assign a first message to be processed by one or more distributed stream processing nodes based on one or more properties of the message. The one or more distributed stream processing nodes can be communicably connected to the distributed storage system and/or the rules engine to provide (1) an answer in response to the first message and/or (2) cause an action to be executed based on the first message.

    Abstract translation: 在一个实施例中,用于处理消息流的扩展策略服务包括分布式流处理计算系统,包括分布式流处理节点,分布式存储系统和规则引擎。 分布式流处理计算系统的流处理引擎可以接收包括请求和/或事件的消息流,并且基于消息的一个或多个属性来分配要由一个或多个分布式流处理节点处理的第一消息。 一个或多个分布式流处理节点可以可通信地连接到分布式存储系统和/或规则引擎,以提供(1)响应于第一消息的答案和/或(2)使得基于 第一条消息。

    CORRECTLY IDENTIFYING POTENTIAL ANOMALIES IN A DISTRIBUTED STORAGE SYSTEM
    36.
    发明申请
    CORRECTLY IDENTIFYING POTENTIAL ANOMALIES IN A DISTRIBUTED STORAGE SYSTEM 有权
    在分布式存储系统中正确识别潜在异常

    公开(公告)号:US20170010931A1

    公开(公告)日:2017-01-12

    申请号:US14794676

    申请日:2015-07-08

    Abstract: A method for assisting evaluation of anomalies in a distributed storage system is disclosed. The method includes a step of monitoring at least one system metric of the distributed storage system. The method further includes steps of maintaining a listing of patterns of the monitored system metric comprising patterns which previously did not result in a failure within one or more nodes of the distributed storage system, and, based on the monitoring, identifying a pattern (i.e., a time series motif) of the monitored system metric as a potential anomaly in the distributed storage system. The method also includes steps of automatically (i.e. without user input) performing a similarity search to determine whether the identified pattern satisfies one or more predefined similarity criteria with at least one pattern of the listing, and, upon positive determination, excepting the identified pattern from being identified as the potential anomaly.

    Abstract translation: 公开了一种用于辅助评估分布式存储系统中的异常的方法。 该方法包括监视分布式存储系统的至少一个系统度量的步骤。 该方法还包括以下步骤:维护所监视的系统度量的模式的列表,其包括先前不会在分布式存储系统的一个或多个节点内导致故障的模式,并且基于该监视,识别模式(即, 监控系统度量的时间序列主题作为分布式存储系统中的潜在异常。 该方法还包括自动执行相似性搜索(即,不进行用户输入)以确定所识别的模式是否满足具有列表的至少一种模式的一个或多个预定义相似性标准的步骤,并且在正确定义之后,除了所识别的模式 被确定为潜在的异常。

    INTERACTIVE MECHANISM TO VIEW LOGS AND METRICS UPON AN ANOMALY IN A DISTRIBUTED STORAGE SYSTEM
    37.
    发明申请
    INTERACTIVE MECHANISM TO VIEW LOGS AND METRICS UPON AN ANOMALY IN A DISTRIBUTED STORAGE SYSTEM 审中-公开
    交互机制查看分布式存储系统中的异常记录和度量

    公开(公告)号:US20170010930A1

    公开(公告)日:2017-01-12

    申请号:US14794650

    申请日:2015-07-08

    Abstract: A method for assisting evaluation of anomalies in a distributed storage system is disclosed. The method includes monitoring at least one system metric of the system and creating a mapping between values and/or patterns of the system metric and one or more services configured to generate logs for the system. The method further includes detecting a potential anomaly in the system based on the monitoring, the potential anomaly being associated with a value and/or a pattern of the monitored system metric. The method also includes using the mapping to identify one or more logs associated with the potential anomaly, displaying a graphical representation of at least a part of monitoring the system metric, the graphical representation indicating the potential anomaly, and providing an overlay over the graphical representation, the overlay comprising an indicator of a number of the logs associated with the potential anomaly.

    Abstract translation: 公开了一种用于辅助评估分布式存储系统中的异常的方法。 该方法包括监视系统的至少一个系统度量并且创建系统度量的值和/或模式之间的映射以及被配置为生成用于系统的日志的一个或多个服务。 该方法还包括基于监视来检测系统中的潜在异常,所述潜在异常与监视的系统度量的值和/或模式相关联。 该方法还包括使用映射来识别与潜在异常相关联的一个或多个日志,显示监视系统度量的至少一部分的图形表示,指示潜在异常的图形表示,以及在图形表示上提供覆盖 ,所述覆盖层包括与所述潜在异常相关联的多个日志的指示符。

    Allocating resources for multi-phase, distributed computing jobs
    38.
    发明授权
    Allocating resources for multi-phase, distributed computing jobs 有权
    为多阶段分布式计算任务分配资源

    公开(公告)号:US09489225B2

    公开(公告)日:2016-11-08

    申请号:US14156149

    申请日:2014-01-15

    CPC classification number: G06F9/45533 G06F9/44505 G06F9/5005

    Abstract: In one embodiment, data indicative of the size of an intermediate data set generated by a first resource device is received at a computing device. The intermediate data set is associated with a virtual machine to process the intermediate data set. A virtual machine configuration is determined based on the size of the intermediate data set. A second resource device is selected to execute the virtual machine based on the virtual machine configuration and on an available bandwidth between the first and second resource devices. The virtual machine is then assigned to the second resource device to process the intermediate data set.

    Abstract translation: 在一个实施例中,在计算设备处接收指示由第一资源设备生成的中间数据集的大小的数据。 中间数据集与虚拟机相关联以处理中间数据集。 基于中间数据集的大小来确定虚拟机配置。 选择第二资源设备以基于虚拟机配置和第一和第二资源设备之间的可用带宽来执行虚拟机。 然后将虚拟机分配给第二资源设备以处理中间数据集。

    ELASTIC SCALE OUT POLICY SERVICE
    39.
    发明申请
    ELASTIC SCALE OUT POLICY SERVICE 有权
    弹性规模服务

    公开(公告)号:US20160182614A1

    公开(公告)日:2016-06-23

    申请号:US14581783

    申请日:2014-12-23

    Abstract: In one embodiment, a scale out policy service for processing a stream of messages includes a distributed stream processing computation system comprising distributed stream processing nodes, a distributed storage system, and a rules engine. A stream processing engine of the distributed stream processing computation system can receive the stream of messages comprising requests and/or events, and assign a first message to be processed by one or more distributed stream processing nodes based on one or more properties of the message. The one or more distributed stream processing nodes can be communicably connected to the distributed storage system and/or the rules engine to provide (1) an answer in response to the first message and/or (2) cause an action to be executed based on the first message.

    Abstract translation: 在一个实施例中,用于处理消息流的扩展策略服务包括分布式流处理计算系统,包括分布式流处理节点,分布式存储系统和规则引擎。 分布式流处理计算系统的流处理引擎可以接收包括请求和/或事件的消息流,并且基于消息的一个或多个属性来分配要由一个或多个分布式流处理节点处理的第一消息。 一个或多个分布式流处理节点可以可通信地连接到分布式存储系统和/或规则引擎,以提供(1)响应于第一消息的答案和/或(2)使得基于 第一条消息。

    Optimized assignments and/or generation virtual machine for reducer tasks
    40.
    发明授权
    Optimized assignments and/or generation virtual machine for reducer tasks 有权
    针对减速机任务优化分配和/或生成虚拟机

    公开(公告)号:US09367344B2

    公开(公告)日:2016-06-14

    申请号:US14509691

    申请日:2014-10-08

    CPC classification number: G06F9/45558 G06F9/5066 G06F2009/45562 H04L47/78

    Abstract: The present disclosure relates to assignment or generation of reducer virtual machines after the “map” phase is substantially complete in MapReduce. Instead of a priori placement, distribution of keys after the “map” phase over the mapper virtual machines can be used to efficiently reducer tasks in virtualized cloud infrastructure like OpenStack. By solving a constraint optimization problem, reducer VMs can be optimally assigned to process keys subject to certain constraints. In particular, the present disclosure describes a special variable matrix. Furthermore, the present disclosure describes several possible cost matrices for representing the costs determined based on the key distribution over the mapper VMs (and other suitable factors).

    Abstract translation: 本公开涉及在MapReduce中的“映射”阶段基本完成之后分配或生成reducer虚拟机。 在映射器虚拟机上的“映射”阶段之后,可以使用OpenStack虚拟化云基础设施中的有效减少任务来代替先验位置分配密钥。 通过解决约束优化问题,可以将reducer VM最优化地分配给具有某些限制的处理密钥。 具体地,本公开描述了特殊变量矩阵。 此外,本公开描述了用于表示基于映射器VM上的密钥分布(和其他合适因素)确定的成本的几种可能的成本矩阵。

Patent Agency Ranking