利索-全球专利检索

  1. 登录
  2. 注册

搜索结果

21 条记录 (耗时 0.019 秒)

    Dynamically performing data processing in a data pipeline system
    4.
    发明授权
    Dynamically performing data processing in a data pipeline system 有权

    公开(公告)号：US10176217B1

    公开(公告)日：2019-01-08

    申请号：US15698574

    申请日：2017-09-07

    申请人： Palantir Technologies, Inc.

    发明人： Hao Dang ,  Gustav Brodman ,  Yi Xue ,  Stacey Milspaw ,  Yifei Huang ,  Yanran Lu

    IPC分类号： G06F17/30 ,  G06F9/455

    摘要： Techniques for automatically scheduling builds of derived datasets in a distributed database system that supports pipelined data transformations are described herein. In an embodiment, a data processing method comprises, in association with a distributed database system that implements one or more data transformation pipelines, each of the data transformation pipelines comprising at least a first dataset, a first transformation, a second derived dataset and dataset dependency and timing metadata, detecting an arrival of a new raw dataset or new derived dataset; in response to the detecting, obtaining from the dataset dependency and timing metadata a dataset subset comprising those datasets that depend on at least the new raw dataset or new derived dataset; for each member dataset in the dataset subset, determining if the member dataset has a dependency on any other dataset that is not yet arrived, and in response to determining that the member dataset does not have a dependency on any other dataset that is not yet arrived: initiating a build of a portion of the data transformation pipeline comprising the member dataset and all other datasets on which the member dataset is dependent, without waiting for arrival of other datasets.

    DYNAMICALLY PERFORMING DATA PROCESSING IN A DATA PIPELINE SYSTEM
    9.
    发明申请
    DYNAMICALLY PERFORMING DATA PROCESSING IN A DATA PIPELINE SYSTEM 审中-公开

    公开(公告)号：US20190114289A1

    公开(公告)日：2019-04-18

    申请号：US16208435

    申请日：2018-12-03

    申请人： Palantir Technologies, Inc.

    发明人： Hao Dang ,  Gustav Brodman ,  Yi Xue ,  Stacey Milspaw ,  Yifei Huang ,  Yanran Lu

    IPC分类号： G06F16/182 ,  G06F9/455

    摘要： Techniques for automatically scheduling builds of derived datasets in a distributed database system that supports pipelined data transformations are described herein. In an embodiment, a data processing method comprises, in association with a distributed database system that implements one or more data transformation pipelines, each of the data transformation pipelines comprising at least a first dataset, a first transformation, a second derived dataset and dataset dependency and timing metadata, detecting an arrival of a new raw dataset or new derived dataset; in response to the detecting, obtaining from the dataset dependency and timing metadata a dataset subset comprising those datasets that depend on at least the new raw dataset or new derived dataset; for each member dataset in the dataset subset, determining if the member dataset has a dependency on any other dataset that is not yet arrived, and in response to determining that the member dataset does not have a dependency on any other dataset that is not yet arrived: initiating a build of a portion of the data transformation pipeline comprising the member dataset and all other datasets on which the member dataset is dependent, without waiting for arrival of other datasets.