Abstract:
Examples herein involve processing data in a distributed data processing system using an off-heap memory store. An example involves allocating a shared memory region of a shared memory to store attributes corresponding to a first partition of a distributed data system, and updating, in the shared memory region, the attributes corresponding to updates to the local data from process iterations of the first partition, such that a second partition of the distributed data system has access to the updated attributes.
Abstract:
In some examples, a system includes a shared memory, a metadata store separate from the shared memory, and a management engine. The management engine may receive input data, partition the input data into multiple data partitions to cache the input data in the shared memory as a distributed data object, send partition store instructions to store the multiple data partitions within the shared memory. The management engine may also obtain partition metadata for the multiple data partitions that form the distributed data object. The partition metadata may include global memory addresses within the shared memory for the multiple data partitions. The management engine may further store the partition metadata in the metadata store.
Abstract:
Examples herein involve fault tolerance in a shared memory. In examples herein, a metadata store of a shared memory indicating versions of data partitions of a resilient distributed dataset and a valid flag for the partitions of the resilient distributed dataset are used to achieve fault tolerance and/or recover from faults in the share memory.
Abstract:
Examples herein involve fault tolerance in a shared memory. In examples herein, a metadata store of a shared memory indicating versions of data partitions of a resilient distributed dataset and a valid flag for the partitions of the resilient distributed dataset are used to achieve fault tolerance and/or recover from faults in the share memory.
Abstract:
Examples relate to deploying distributed applications using virtual nodes. In some examples, virtual nodes are created and are each assigned a core subset of a number of processing cores, an Internet protocol (IP) address, and an in-memory file system configured to provide access to a portion of physically shared memory. At this stage, a distributed application that is configured to be deployed to a plurality of machine nodes is deployed to the plurality of virtual nodes. On a first virtual node, a reference to a first dataset stored in physically shared memory is sent to a second virtual node, where the physically shared memory is accessible to each of the plurality of virtual nodes. Next, on the second virtual node, the first dataset is accessed through the in-memory file system of the first virtual node.
Abstract:
Examples herein involve processing data in a distributed data processing system using an off-heap memory store. An example involves allocating a shared memory region of a shared memory to store attributes corresponding to a first partition of a distributed data system, and updating, in the shared memory region, the attributes corresponding to updates to the local data from process iterations of the first partition, such that a second partition of the distributed data system has access to the updated attributes.
Abstract:
Examples relate to deploying distributed applications using virtual nodes. In some examples, virtual nodes are created and are each assigned a core subset of a number of processing cores, an Internet protocol (IP) address, and an in-memory file system configured to provide access to a portion of physically shared memory. At this stage, a distributed application that is configured to be deployed to a plurality of machine nodes is deployed to the plurality of virtual nodes. On a first virtual node, a reference to a first dataset stored in physically shared memory is sent to a second virtual node, where the physically shared memory is accessible to each of the plurality of virtual nodes. Next, on the second virtual node, the first dataset is accessed through the in-memory file system of the first virtual node.
Abstract:
An example technique includes assigning partitions of a dataset of multidimensional points to a plurality of local memory nodes of a multicore machine and using the local memory nodes for a search query to determine similarity matches in the dataset for a given multidimensional point. The using includes parallel searching with the local memory nodes in the assigned partitions to identify candidate similarity matches to the given multidimensional point using indexes derived from the multidimensional points, the parallel searching for each node progressing through a sequence of search distances and providing an ongoing search result for each search distance from the given multidimensional point and regulating an extent of the parallel searching based on the ongoing search results.
Abstract:
An example technique includes assigning partitions of a dataset of multidimensional points to a plurality of local memory nodes of a multicore machine and using the local memory nodes for a search query to determine similarity matches in the dataset for a given multidimensional point. The using includes parallel searching with the local memory nodes in the assigned partitions to identify candidate similarity matches to the given multidimensional point using indexes derived from the multidimensional points, the parallel searching for each node progressing through a sequence of search distances and providing an ongoing search result for each search distance from the given multidimensional point and regulating an extent of the parallel searching based on the ongoing search results.
Abstract:
Storing time series data for a search query includes identifying a time series whose representation is to be pre-computed based on available memory storage, pre-computing at least one representation of the identified time series, and storing the at least one representation in the memory storage.