公开(公告)号：US11366797B2

公开(公告)日：2022-06-21

申请号：US17134862

申请日：2020-12-28

Applicant: Google LLC

Inventor： Jeffrey Dean ,  Sanjay Ghemawat

IPC: G06F16/22 ,  G06F16/23 ,  G06F16/2453 ,  G06F9/48 ,  G06F9/54

Abstract: A method performs large-scale data processing in a distributed and parallel processing environment. The method defines application-independent map and reduce operations, each invoking one or more library functions that automatically handle data partitioning, parallelization of computations, and fault tolerance. A user specifies a map operation, which calls one or more of the application-independent map operators to perform data read and write operations. A user also specifies a reduce operation, which calls one or more of the application-independent reduce operators to perform data read and write operations. The method executes application-independent map worker processes. Each map worker process executes the user-specified map operation to read designated portions of input files and store intermediate data values in intermediate data structures. The method also executes application-independent reduce worker processes. Each reduce worker process executes the user-specified reduce operation to read intermediate data values from the intermediate data structures and produce final output data.

公开(公告)号：US20190243804A1

公开(公告)日：2019-08-08

申请号：US16268149

申请日：2019-02-05

Applicant: Google LLC

Inventor： Yasushi Saito ,  Sanjay Ghemawat ,  Jeffrey A. Dean

IPC: G06F16/16 ,  G06F16/174 ,  G06F16/11 ,  G06F16/182 ,  G06F16/215

CPC classification number: G06F16/162 ,  G06F16/11 ,  G06F16/1748 ,  G06F16/182 ,  G06F16/215

Abstract: A method for deleting obsolete files from a file system is provided. The method includes receiving a request to delete a reference to a first target file of a plurality of target files stored in a file system, the first target file having a first target file name. A first reference file whose file name includes the first target file name is identified. The first reference file is deleted from the file system. The method further includes determining whether the file system includes at least one reference file, distinct from the first reference file, whose file name includes the first target file name. In accordance with a determination that the file system does not include the at least one reference file, the first target file is deleted from the file system.

公开(公告)号：US20180247197A1

公开(公告)日：2018-08-30

申请号：US15965742

申请日：2018-04-27

Applicant: Google LLC

Inventor： Paul A. Tucker ,  Jeffrey Adgate Dean ,  Sanjay Ghemawat ,  Yuan Yu

IPC: G06N3/08 ,  G06N5/04

CPC classification number: G06N3/08 ,  G06F9/5038 ,  G06F9/5066 ,  G06N3/0454 ,  G06N3/063 ,  G06N3/084 ,  G06N5/048

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for receiving a request from a client to process a computational graph; obtaining data representing the computational graph, the computational graph comprising a plurality of nodes and directed edges, wherein each node represents a respective operation, wherein each directed edge connects a respective first node to a respective second node that represents an operation that receives, as input, an output of an operation represented by the respective first node; identifying a plurality of available devices for performing the requested operation; partitioning the computational graph into a plurality of subgraphs, each subgraph comprising one or more nodes in the computational graph; and assigning, for each subgraph, the operations represented by the one or more nodes in the subgraph to a respective available device in the plurality of available devices for operation.

公开(公告)号：US12056089B2

公开(公告)日：2024-08-06

申请号：US18239475

申请日：2023-08-29

Applicant: Google LLC

Inventor： Yasushi Saito ,  Sanjay Ghemawat ,  Jeffrey Adgate Dean

IPC: G06F16/16 ,  G06F16/11 ,  G06F16/174 ,  G06F16/182 ,  G06F16/215

CPC classification number: G06F16/162 ,  G06F16/11 ,  G06F16/1748 ,  G06F16/182 ,  G06F16/215

Abstract: A method for deleting obsolete files from a file system is provided. The method includes receiving a request to delete a reference to a first target file of a plurality of target files stored in a file system, the first target file having a first target file name. A first reference file whose file name includes the first target file name is identified. The first reference file is deleted from the file system. The method further includes determining whether the file system includes at least one reference file, distinct from the first reference file, whose file name includes the first target file name. In accordance with a determination that the file system does not include the at least one reference file, the first target file is deleted from the file system.

公开(公告)号：US11556381B2

公开(公告)日：2023-01-17

申请号：US17738909

申请日：2022-05-06

Applicant: Google LLC

Inventor： Jeffrey Adgate Dean ,  Sudip Roy ,  Michael Acheson Isard ,  Aakanksha Chowdhery ,  Brennan Saeta ,  Chandramohan Amyangot Thekkath ,  Daniel William Hurt ,  Hyeontaek Lim ,  Laurent El Shafey ,  Parker Edward Schuh ,  Paul Ronald Barham ,  Ruoming Pang ,  Ryan Sepassi ,  Sanjay Ghemawat ,  Yonghui Wu

IPC: G06F15/82 ,  G06F9/48 ,  G06N3/08 ,  G06N3/063

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributing machine learning workloads, e.g., computations for training a neural network or computing an inference using a neural network, across multiple hardware accelerators. One of the systems comprises a plurality of accelerator islands, each hardware accelerator island comprising a respective plurality of hardware devices that include a plurality of hardware accelerators and a corresponding host for each of the plurality of hardware accelerators; and a respective scheduler for each of the accelerator islands that is configured to schedule workloads across the plurality of accelerators and corresponding hosts in the accelerator island, wherein the system is configured to: receive data representing a machine learning workload; and assign a respective portion of the machine learning workload to each of the plurality of accelerator islands for scheduling by the respective scheduler for the accelerator island.

公开(公告)号：US20220171781A1

公开(公告)日：2022-06-02

申请号：US17673049

申请日：2022-02-16

Applicant: Google LLC

Inventor： Robert C. Pike ,  Sean Quinlan ,  Sean M. Dorward ,  Jeffrey Dean ,  Sanjay Ghemawat

IPC: G06F16/2455 ,  G06F16/28 ,  G06F16/2458 ,  G06F11/14 ,  G06F16/18

Abstract: Systems and methods for analyzing input data records are provided in which a master process initiates a plurality of concurrent first processes each of which comprises, for each data record in at least a subset of a plurality of input data records, creating a parsed representation of the data record and independently applying a procedural language query to the parsed representation to extract one or more values. A respective emit operator is applied to at least one of the extracted one or more values thereby adding corresponding information to a respective intermediate data structure. The respective emit operator implements one of a predefined set of statistical information processing functions. The master process also initiates a plurality of second processes each of which aggregates information from a corresponding subset of intermediate data structures to produce aggregated data that is, in turn, combined to produce output data.

公开(公告)号：US20220019896A1

公开(公告)日：2022-01-20

申请号：US17392690

申请日：2021-08-03

Applicant: Google LLC

Inventor： Vijay Vasudevan ,  Jeffrey Adgate Dean ,  Sanjay Ghemawat

IPC: G06N3/08 ,  G06F9/50 ,  G06F9/54 ,  G06N3/063 ,  G06N3/04 ,  G06F17/16

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for modifying a computational graph to include send and receive nodes. Communication between unique devices performing operations of different subgraphs of the computational graph can be handled efficiently by inserting send and receive nodes into each subgraph. When executed, the operations that these send and receive nodes represent may enable pairs of unique devices to conduct communication with each other in a self-sufficient manner. This shifts the burden of coordinating communication away from the backend, which affords the system that processes this computational graph representation the opportunity to perform one or more other processes while devices are executing subgraphs.

公开(公告)号：US10885012B2

公开(公告)日：2021-01-05

申请号：US16417126

申请日：2019-05-20

Applicant: Google LLC

Inventor： Jeffrey Dean ,  Sanjay Ghemawat

IPC: G06F16/22 ,  G06F16/23 ,  G06F16/2453 ,  G06F9/48 ,  G06F9/54

Abstract: A method performs large-scale data processing in a distributed and parallel processing environment. The method defines application-independent map and reduce operations, each invoking one or more library functions that automatically handle data partitioning, parallelization of computations, and fault tolerance. A user specifies a map operation, which calls one or more of the application-independent map operators to perform data read and write operations. A user also specifies a reduce operation, which calls one or more of the application-independent reduce operators to perform data read and write operations. The method executes application-independent map worker processes. Each map worker process executes the user-specified map operation to read designated portions of input files and store intermediate data values in intermediate data structures. The method also executes application-independent reduce worker processes. Each reduce worker process executes the user-specified reduce operation to read intermediate data values from the intermediate data structures and produce final output data.

公开(公告)号：US10545970B1

公开(公告)日：2020-01-28

申请号：US15656702

申请日：2017-07-21

Applicant: Google LLC

Inventor： Jeffrey Adgate Dean ,  Sanjay Ghemawat ,  Sachin Jain ,  Boris Mazniker

IPC: G06F16/2457 ,  H04L12/18

Abstract: A system includes: an engaging post identifier for identifying and retrieving engaging posts; an extended network post identifier for identifying extended posts from an extended network; a combining module for creating a combined list of added posts from the engaging post and the extended posts, the combining module generating one or more ranked posts by ranking the list of added posts by relevance to a user; and a user interface module for providing the one or more ranked posts. The disclosure also includes a method for finding and providing engaging posts that includes determining engaging posts; determining extended posts from an extended social network using a social graph of the user; adding the engaging posts and the extended posts to create a combined list of added posts; ranking the added posts by relevance to a user; and providing one or more of the ranked posts.

公开(公告)号：US20180173722A1

公开(公告)日：2018-06-21

申请号：US15868928

申请日：2018-01-11

Applicant: Google LLC

Inventor： Jeffrey A. Dean ,  Sanjay Ghemawat ,  Andrew B. Fikes ,  Yasushi Saito

IPC: G06F17/30 ,  G06F9/50

Abstract: A method of accessing data includes storing a table that includes a plurality of tablets corresponding to distinct non-overlapping table portions. Respective pluralities of tablet access objects and application objects are stored in a plurality of servers. A distinct application object and distinct tablet are associated with each tablet access object. Each application object corresponds to a distinct instantiation of an application associated with the table. The tablet access objects and associated application objects are redistributed among the servers in accordance with a first load-balancing criterion. A first request directed to a respective tablet is received from a client. In response, the tablet access object associated with the respective tablet is used to perform a data access operation on the respective tablet, and the application object associated with the respective tablet is used to perform an additional computational operation to produce a result to be returned to the client.