Task packing scheduling process for long running applications

    公开(公告)号:US10733024B2

    公开(公告)日:2020-08-04

    申请号:US15988535

    申请日:2018-05-24

    IPC分类号: G06F9/50 G06F9/48

    摘要: In general, the invention is directed to systems and methods of distributing tasks amongst servers or nodes in a cluster in a cloud-based big data environment, including: establishing a high_server_threshold; dividing active servers/nodes into at least three (3) categories of high usage servers, comprising servers on which usage is greater than the high_server_threshold; medium usage servers, comprising servers on which usage is less than the high_server_threshold, but is greater than zero; and low usage servers, comprising servers that are currently not utilized; receiving one or more tasks to be performed; scheduling the tasks by: first requesting that medium usage servers take tasks; if tasks remain that are not scheduled on the medium usage servers, schedule remaining tasks on low usage servers; if any tasks remain that are not scheduled on medium usage servers or low usage servers, scheduling remaining tasks on high usage servers.

    Pure-Spot and Dynamically Rebalanced Auto-Scaling Clusters
    2.
    发明申请
    Pure-Spot and Dynamically Rebalanced Auto-Scaling Clusters 审中-公开
    纯点和动态重新平衡自动缩放集群

    公开(公告)号:US20160358249A1

    公开(公告)日:2016-12-08

    申请号:US15176061

    申请日:2016-06-07

    IPC分类号: G06Q30/08 H04L29/08

    CPC分类号: G06Q30/08 H04L67/10

    摘要: The present invention is generally directed to systems and methods of providing automatic scaling pure-spot clusters. Such dusters may be dynamically rebalanced for further costs savings. In accordance with some methods of the present invention may include a method of utilizing a cluster in a big data cloud computing environment where instances may include reserved on-demand instances for a set price and on-demand spot instances that may be bid on by a user, the method including: creating one or more stable nodes, comprising spot instances with a bid price above a price for an equivalent on-demand instance; creating one or more volatile nodes, comprising spot instances with a bid price below a price for an equivalent on-demand instance; using one or more of the stable nodes as a master node; and using the volatile nodes as slave nodes.

    摘要翻译: 本发明一般涉及提供自动缩放纯点簇的系统和方法。 这种除尘器可以动态地重新平衡以进一步节省成本。 根据本发明的一些方法可以包括在大数据云计算环境中利用集群的方法,其中实例可以包括针对设定价格的保留按需实例以及可以通过一个 用户,所述方法包括:创建一个或多个稳定节点,其包括具有高于等价按需实例的价格的投标价格的现货实体; 创建一个或多个易失性节点,其包括具有低于等价按需实例的价格的投标价格的现货实例; 使用一个或多个稳定节点作为主节点; 并使用易失性节点作为从节点。

    System and method for transferring volume ownership in networked storage
    3.
    发明授权
    System and method for transferring volume ownership in networked storage 有权
    网络存储中传输卷所有权的系统和方法

    公开(公告)号:US07788361B2

    公开(公告)日:2010-08-31

    申请号:US11867130

    申请日:2007-10-04

    IPC分类号: G06F15/173

    摘要: In a method for operating a data storage system, a request is issued that an ownership attribute of a logical arrangement of storage space associated with a first storage system be associated with a second storage system. In response to the request, the ownership attribute is changed to a state of unowned. In response to the request, and in response to the state of unowned, the ownership attribute of the logical arrangement of storage space is changed to a new set of attributes associated with the second storage system to change ownership of the logical arrangement of storage space from the first storage system to the second storage system.

    摘要翻译: 在用于操作数据存储系统的方法中,发出与第一存储系统相关联的存储空间的逻辑布置的所有权属性与第二存储系统相关联的请求。 响应请求,所有权属性更改为未知状态。 响应于该请求,并且响应于未知状态,将存储空间的逻辑布置的所有权属性改变为与第二存储系统相关联的新的属性集合,以将存储空间的逻辑布置的所有权从 第一个存储系统到第二个存储系统。

    Systems and method of implementing disk ownership in networked storage
    4.
    发明授权
    Systems and method of implementing disk ownership in networked storage 有权
    在网络存储中实现磁盘所有权的系统和方法

    公开(公告)号:US07650412B2

    公开(公告)日:2010-01-19

    申请号:US10027457

    申请日:2001-12-21

    IPC分类号: G06F15/16

    摘要: A system and method for disk ownership in a network storage system. Each disk has two ownership attributes set to show that a particular file server owns the disk. In a preferred embodiment the first ownership attribute is the serial number of the file server being written to a specific location on each disk and the second ownership attribute is setting a SCSI-3 persistent reservation. In a system utilizing this disk ownership method, multiple file servers can read data from a given disk, but only the file server that owns a particular disk can write data to the disk.

    摘要翻译: 一种网络存储系统中磁盘所有权的系统和方法。 每个磁盘都有两个所有权属性,用于显示特定的文件服务器拥有该磁盘。 在优选实施例中,第一所有权属性是要写入每个磁盘上特定位置的文件服务器的序列号,而第二所有权属性正在设置SCSI-3永久保留。 在使用此磁盘所有权方法的系统中,多个文件服务器可以从给定磁盘读取数据,但只有拥有特定磁盘的文件服务器才能将数据写入磁盘。

    System and method for establishing a peer connection using reliable RDMA primitives
    5.
    发明授权
    System and method for establishing a peer connection using reliable RDMA primitives 有权
    使用可靠的RDMA原语建立对等连接的系统和方法

    公开(公告)号:US07593996B2

    公开(公告)日:2009-09-22

    申请号:US10622565

    申请日:2003-07-18

    IPC分类号: G06F15/167

    CPC分类号: H04L29/06 H04L69/16

    摘要: A system and method for reliable peer-to-peer communication is provided. The system and method includes a cluster connection manager that acts as a unified resource for opening, closing and maintaining peer-to-peer communication channels between applications executing on storage systems comprising a storage system cluster. The cluster connection manager communicates with cluster connection manager clients to maintain and manage the communication channels. The cluster connection manager may also be operatively interconnected with a plurality of cluster interconnect devices, thereby enabling failover operation in the event that a cluster interconnect device suffers an error condition.

    摘要翻译: 提供了一种用于可靠的对等通信的系统和方法。 该系统和方法包括一个集群连接管理器,其作为用于在包括存储系统集群的存储系统上执行的应用之间打开,关闭和维护对等通信信道的统一资源。 集群连接管理器与集群连接管理器客户端进行通信,以维护和管理通信通道。 集群连接管理器还可以与多个集群互连设备可操作地互连,从而在集群互连设备遇到错误状况的情况下实现故障切换操作。

    Feature Discretization and Cardinality Reduction Using Collaborative Filtering Techniques
    6.
    发明申请
    Feature Discretization and Cardinality Reduction Using Collaborative Filtering Techniques 审中-公开
    使用协同过滤技术进行功能离散化和基数减少

    公开(公告)号:US20090077093A1

    公开(公告)日:2009-03-19

    申请号:US11857787

    申请日:2007-09-19

    IPC分类号: G06F17/30

    CPC分类号: G06F16/217

    摘要: A system and method to perform discretization and cardinality reduction of item attributes using collaborative filtering techniques are described. Data input by a user is received over a network, the input data further including a plurality of items and associated item metadata related to events performed by the user. The input data is further processed to obtain a predetermined number of groupings, each grouping having a calculated value based on a distance parameter between corresponding attributes of each item stored within the item metadata. Finally, a similarity parameter is computed between each pair of items within the plurality of items based on associated groupings and recommendations of the items are presented to the user based on the corresponding calculated similarity parameter.

    摘要翻译: 描述了使用协同过滤技术来执行项目属性的离散化和基数减少的系统和方法。 通过网络接收用户输入的数据,输入数据还包括与由用户执行的事件相关的多个项目和相关联的项目元数据。 进一步处理输入数据以获得预定数量的分组,每个分组具有基于存储在项目元数据中的每个项目的对应属性之间的距离参数的计算值。 最后,基于相关联的分组,在多个项目中的每对项目之间计算相似性参数,并且基于相应的计算出的相似度参数向用户呈现建议。

    Attribute-Based Item Similarity Using Collaborative Filtering Techniques
    7.
    发明申请
    Attribute-Based Item Similarity Using Collaborative Filtering Techniques 审中-公开
    基于属性的项目相似性使用协同过滤技术

    公开(公告)号:US20090077081A1

    公开(公告)日:2009-03-19

    申请号:US11857847

    申请日:2007-09-19

    IPC分类号: G06F7/00

    摘要: A system and method to compute attribute-based similarity between items using collaborative filtering techniques are described. Events input by a user over a network are received, the events further including a plurality of items and associated item metadata. A similarity value is further computed between each pair of items within the plurality of stems based on corresponding attributes of each item stored within the item metadata. Finally, recommendations of the items to the user are presented based on the corresponding calculated similarity value.

    摘要翻译: 描述了使用协同过滤技术来计算项目之间基于属性的相似性的系统和方法。 接收由用户通过网络输入的事件,所述事件还包括多个项目和相关联的项目元数据。 基于存储在项目元数据中的每个项目的相应属性,进一步在多个词干中的每对项目之间计算相似度值。 最后,基于对应的计算出的相似度值来呈现用户的项目的建议。

    TASK PACKING SCHEDULING PROCESS FOR LONG RUNNING APPLICATIONS

    公开(公告)号:US20180341524A1

    公开(公告)日:2018-11-29

    申请号:US15988535

    申请日:2018-05-24

    IPC分类号: G06F9/50 G06F9/48

    摘要: In general, the invention is directed to systems and methods of distributing tasks amongst servers or nodes in a cluster in a cloud-based big data environment, including: establishing a high_server_threshold; dividing active servers/nodes into at least three (3) categories of high usage servers, comprising servers on which usage is greater than the high_server_threshold; medium usage servers, comprising servers on which usage is less than the high_server_threshold, but is greater than zero; and low usage servers, comprising servers that are currently not utilized; receiving one or more tasks to be performed; scheduling the tasks by: first requesting that medium usage servers take tasks; if tasks remain that are not scheduled on the medium usage servers, schedule remaining tasks on low usage servers; if any tasks remain that are not scheduled on medium usage servers or low usage servers, scheduling remaining tasks on high usage servers.

    High Performance Hadoop with New Generation Instances
    9.
    发明申请
    High Performance Hadoop with New Generation Instances 审中-公开
    具有新一代实例的高性能Hadoop

    公开(公告)号:US20160117107A1

    公开(公告)日:2016-04-28

    申请号:US14920640

    申请日:2015-10-22

    IPC分类号: G06F3/06

    摘要: The present invention is generally directed to a distributed computing system comprising a plurality of computational clusters, each computational cluster comprising a plurality of compute optimized instances, each instance comprising local instance data storage and in communication with reserved disk storage, wherein processing hierarchy provides priority to local instance data storage before providing priority to reserved disk storage.

    摘要翻译: 本发明一般涉及包括多个计算集群的分布式计算系统,每个计算集群包括多个计算优化实例,每个实例包括本地实例数据存储并与保留磁盘存储器通信,其中处理层次结构提供优先级 本地实例数据存储,然后提供优先级到保留的磁盘存储。

    System and method for coordinating cluster state information
    10.
    发明授权
    System and method for coordinating cluster state information 有权
    用于协调集群状态信息的系统和方法

    公开(公告)号:US07953924B1

    公开(公告)日:2011-05-31

    申请号:US12692030

    申请日:2010-01-22

    IPC分类号: G06F12/00 G06F15/167

    摘要: A method for managing a plurality of servers is disclosed. Each server of the plurality of servers has access to data stored by other servers. The data is stored to one or more data storage devices. Coordinating information is written for the plurality of servers to a master mailbox record. The coordinating information includes data that each server uses to recover after a failure by a server. The master mailbox record is stored on a selected storage device at a location known to the plurality of servers, and the selected storage device is designated as a lock storage device. A plurality of lock storage devices is chosen so that in the event of failure of a server of the plurality of servers, at least one lock storage device will be available to the remaining servers.

    摘要翻译: 公开了一种用于管理多个服务器的方法。 多个服务器中的每个服务器可以访问由其他服务器存储的数据。 数据被存储到一个或多个数据存储设备。 将多个服务器的协调信息写入主邮箱记录。 协调信息包括每台服务器在服务器发生故障后用于恢复的数据。 主信箱记录被存储在多个服务器已知的位置上的所选存储设备上,并且所选择的存储设备被指定为锁定存储设备。 选择多个锁存储设备,使得在多个服务器的服务器发生故障的情况下,至少一个锁存储设备将可用于剩余的服务器。