公开(公告)号：US20250131005A1

公开(公告)日：2025-04-24

申请号：US18885559

申请日：2024-09-13

Applicant: Beijing Volcano Engine Technology Co., Ltd.

Inventor： Kejian JU ,  Leilei HU ,  Junling DU

IPC: G06F16/25

Abstract: Embodiments of the present disclosure provide a data processing method, apparatus based on data Lake and data Warehouse integration, and an electronic device. The method includes: pre-storing at least two protocol files; receiving a first metadata acquisition request; determining a target protocol file for processing the first metadata acquisition request from the at least two protocol files according to an engine type of the data processing engine which sends the first metadata acquisition request; and parsing, based on the target protocol file, the first metadata acquisition request and acquiring metadata corresponding to the first metadata acquisition request from the metadata storage space. Therefore, for at least two sets of external protocols, one set of metadata is stored, accordingly, the data storage capacity can be reduced; with the setting of one set of metadata, the metadata accessing rights management can be unified, and the security of data can be improved.

公开(公告)号：US20250124032A1

公开(公告)日：2025-04-17

申请号：US18827533

申请日：2024-09-06

Applicant: Beijing Volcano Engine Technology Co., Ltd.

Inventor： Leilei HU ,  Kejian JU ,  Zhaowei HUANG ,  Junling DU

IPC: G06F16/2453 ,  G06F16/22 ,  G06F16/23 ,  G06F16/25

Abstract: A data reading and writing method, apparatus and electronic device are provided. In one embodiment, the method includes: receiving a data reading and writing request for a first database, wherein the first database stores data in a column-oriented storage format, the data reading and writing request is used for reading/writing data from/into a row storage table, the first database includes a front-end node and a back-end node, the front-end node is used for receiving the data reading and writing request and the back-end node is used for storing data; parsing the data reading and writing request in the front-end node and sending a parsing result to the back-end node; and reading/writing data from/into a second database by using the parsing result in the back-end node, wherein the first database calls data from the second database, and the second database stores data in a row-oriented storage format.

公开(公告)号：US20240095246A1

公开(公告)日：2024-03-21

申请号：US18470184

申请日：2023-09-19

Applicant: BEIJING VOLCANO ENGINE TECHNOLOGY CO., LTD

Inventor： Zhaowei HUANG ,  Leilei HU ,  Junling DU ,  Xuan LUO

IPC: G06F16/2455 ,  G06F16/21 ,  G06F16/242

CPC classification number: G06F16/24552 ,  G06F16/219 ,  G06F16/244

Abstract: A data query method based on Doris includes: acquiring history data in a bottom operator in a storage engine architecture of the Doris, and performing an aggregation calculation on the history data to obtain an aggregation result of the history data; caching, by a preset cache system, the aggregation result of the history data; performing an aggregation calculation on real-time data in a newly inserted bottom operator in the Doris, in response to a data query request, to obtain an aggregate result of the real-time data; acquiring the aggregation result of the history data from the preset cache system, and merging the aggregation result of the history data with the aggregation result of the real-time data to obtain merged data; and sending, by a data sending operator in the Doris, the merged data to an upper layer of an execution engine, to output a final data query result.