Dynamic Computational Resource Allocation

    公开(公告)号:US20220391257A1

    公开(公告)日:2022-12-08

    申请号:US17831978

    申请日:2022-06-03

    Abstract: A method for performing a distributed computation on a computing system using computational resources dynamically allocated using a computational resource manager includes storing information specifying quantities of computational resources associated with respective ones of a number of program portions of the program, where the program portions perform successive transformations of data and each program portion uses computational resources granted by the computational resource manager enabling computation associated with that program portion to be performed in the computing system, requesting a first quantity of computational resources associated with a first program portion of the number of program portions from the computational resource manager, receiving a second quantity of computational resources from the computational resource manager, less than the requested first quantity of computational resources, performing computation associated with the first portion of the program using the second quantity of computational resources, while performing the computation associated with the first portion of the program using the second quantity of computational resources, receiving an additional quantity of computational resources from the computational resource manager, and performing an additional computation associated with the first portion of the program using the additional quantity of computational resources while performing the computation associated with the first portion using the second quantity of computational resources.

    Compiling graph-based program specifications

    公开(公告)号:US11301445B2

    公开(公告)日:2022-04-12

    申请号:US16701193

    申请日:2019-12-03

    Abstract: A graph-based program specification includes: a plurality of components, each corresponding to a processing task and including one or more ports for sending or receiving one or more data elements; and one or more links, each connecting an output port of an upstream component of the plurality of components to an input port of a downstream component of the plurality of components. Prepared code is generated representing subsets of the plurality of components, including: identifying a plurality of subset boundaries between components in different subsets based at least in part on characteristics of linked components; forming the subsets based on the identified subset boundaries; and generating prepared code for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in that formed subset to be performed according to information embedded in the prepared code for that formed subset.

    Specifying components in graph-based programs

    公开(公告)号:US10896025B2

    公开(公告)日:2021-01-19

    申请号:US16214240

    申请日:2018-12-10

    Abstract: User input is received specifying components of a graph-based program specification. User input is received specifying links, at least some connecting an output port of an upstream component to an input port of a downstream component. The graph-based program specification is processed to identify one or more subsets of the components, including: identifying one or more subset entry points and one or more subset exit points that occur between components in different subsets based at least in part on data processing characteristics of linked components, and forming the subsets based on the identified subset entry points and exit points. A visual representation of the formed subsets is rendered within a user interface. Prepared code is generated for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in each formed subset to be performed.

    Executable logic for processing keyed data in networks

    公开(公告)号:US10579753B2

    公开(公告)日:2020-03-03

    申请号:US15376129

    申请日:2016-12-12

    Abstract: A method implemented by a data processing system for processing data items of a stream of data items, including: accessing a specification that represents the executable logic, wherein a state of the specification for a particular value of the key specifies one or more portions of the executable logic that are executable in that state; receiving, over an input device or port, data items of a stream of data; for a first one of the data items of the stream, identifying a first state of the specification for a value of the key associated with that first one of the data items; processing, by the data processing system, the first one of the data items according to one or more portions of executable logic that are represented in the specification as being associated with the first state.

    Managing a computing cluster using time interval counters

    公开(公告)号:US10558493B2

    公开(公告)日:2020-02-11

    申请号:US16175133

    申请日:2018-10-30

    Abstract: A method for processing state update requests in a distributed data processing system with a number of processing nodes includes maintaining a number of counters including a working counter indicating a current time interval, a replication counter indicating a time interval for which all requests associated with that time interval are replicated at multiple processing nodes of the number of processing nodes, and a persistence counter indicating a time interval of the number of time intervals for which all requests associated with that time interval are stored in persistent storage. The counters are used to manage processing of the state update requests.

    SPECIFYING COMPONENTS IN GRAPH-BASED PROGRAMS

    公开(公告)号:US20190354348A1

    公开(公告)日:2019-11-21

    申请号:US16214240

    申请日:2018-12-10

    Abstract: User input is received specifying components of a graph-based program specification. User input is received specifying links, at least some connecting an output port of an upstream component to an input port of a downstream component. The graph-based program specification is processed to identify one or more subsets of the components, including: identifying one or more subset entry points and one or more subset exit points that occur between components in different subsets based at least in part on data processing characteristics of linked components, and forming the subsets based on the identified subset entry points and exit points. A visual representation of the formed subsets is rendered within a user interface. Prepared code is generated for each formed subset that when used for execution by a runtime system causes processing tasks corresponding to the components in each formed subset to be performed.

    Data processing graph compilation

    公开(公告)号:US10423395B2

    公开(公告)日:2019-09-24

    申请号:US16042205

    申请日:2018-07-23

    Abstract: A received graph-based program specification includes: a plurality of components, each corresponding to at least one operation; and a plurality of directed links each connecting an upstream component to a downstream component. Processed code is generated representing one or more groups of operations by: identifying a possible level of concurrency in a first group of operations based at least in part on a topology of the graph, such that multiple operations in the first group are not prevented by the topology of the graph from executing concurrently; analyzing at least some of the operations in the first group to determine runtime characteristics associated with the analyzed operations; and generating processed code for executing the operations, where the processed code enforces a reduced level of concurrency in the first group, lower than the identified possible level of concurrency, based at least in part on the determined runtime characteristics.

    Mapping attributes of keyed entities

    公开(公告)号:US10191863B2

    公开(公告)日:2019-01-29

    申请号:US14658440

    申请日:2015-03-16

    Abstract: One or more mappings each define a correspondence between input attributes of an input entity and output attributes of an output entity, where the input out output entities each include one or more key attributes identified as part of a unique key. Computing result information, displayed in a user interface, includes: processing instances of a first input entity to generate instances of a first output entity; determining one or more mapped input attributes of the first input entity that correspond to each of the key attributes of the first output entity; generating the instances of the first output entity based on the determined one or more mapped input attributes; computing a total number of instances of the first input entity that were processed; and computing a total number of instances of the first output entity that were generated.

Patent Agency Ranking