公开(公告)号：US12067491B2

公开(公告)日：2024-08-20

申请号：US18131567

申请日：2023-04-06

Applicant: DeepMind Technologies Limited

Inventor： David Silver ,  Oriol Vinyals ,  Maxwell Elliot Jaderberg

IPC: G06N20/00 ,  G06F18/214 ,  G06N3/08 ,  H04L9/40

CPC classification number: G06N3/08 ,  G06F18/214 ,  H04L63/205

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a policy neural network having a plurality of policy parameters and used to select actions to be performed by an agent to control the agent to perform a particular task while interacting with one or more other agents in an environment. In one aspect, the method includes: maintaining data specifying a pool of candidate action selection policies; maintaining data specifying respective matchmaking policy; and training the policy neural network using a reinforcement learning technique to update the policy parameters. The policy parameters define policies to be used in controlling the agent to perform the particular task.

公开(公告)号：US20240185070A1

公开(公告)日：2024-06-06

申请号：US18528640

申请日：2023-12-04

Applicant: DeepMind Technologies Limited

Inventor： Karen Simonyan ,  David Silver ,  Julian Schrittwieser

IPC: G06N3/08 ,  G06N7/01

CPC classification number: G06N3/08 ,  G06N7/01

Abstract: Methods, systems and apparatus, including computer programs encoded on computer storage media, for training an action selection neural network. One of the methods includes receiving an observation characterizing a current state of the environment; determining a target network output for the observation by performing a look ahead search of possible future states of the environment starting from the current state until the environment reaches a possible future state that satisfies one or more termination criteria, wherein the look ahead search is guided by the neural network in accordance with current values of the network parameters; selecting an action to be performed by the agent in response to the observation using the target network output generated by performing the look ahead search; and storing, in an exploration history data store, the target network output in association with the observation for use in updating the current values of the network parameters.

公开(公告)号：US11842281B2

公开(公告)日：2023-12-12

申请号：US17183618

申请日：2021-02-24

Applicant: DeepMind Technologies Limited

Inventor： Volodymyr Mnih ,  Wojciech Czarnecki ,  Maxwell Elliot Jaderberg ,  Tom Schaul ,  David Silver ,  Koray Kavukcuoglu

IPC: G06N3/08 ,  G06N20/00 ,  G06N3/00 ,  G06N3/04 ,  G06N3/084 ,  G06N3/006 ,  G06N3/044 ,  G06N3/045

CPC classification number: G06N3/084 ,  G06N3/006 ,  G06N3/044 ,  G06N3/045 ,  G06N20/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a reinforcement learning system. The method includes: training an action selection policy neural network, and during the training of the action selection neural network, training one or more auxiliary control neural networks and a reward prediction neural network. Each of the auxiliary control neural networks is configured to receive a respective intermediate output generated by the action selection policy neural network and generate a policy output for a corresponding auxiliary control task. The reward prediction neural network is configured to receive one or more intermediate outputs generated by the action selection policy neural network and generate a corresponding predicted reward. Training each of the auxiliary control neural networks and the reward prediction neural network comprises adjusting values of the respective auxiliary control parameters, reward prediction parameters, and the action selection policy network parameters.

公开(公告)号：US11803750B2

公开(公告)日：2023-10-31

申请号：US17019927

申请日：2020-09-14

Applicant: DeepMind Technologies Limited

Inventor： Timothy Paul Lillicrap ,  Jonathan James Hunt ,  Alexander Pritzel ,  Nicolas Manfred Otto Heess ,  Tom Erez ,  Yuval Tassa ,  David Silver ,  Daniel Pieter Wierstra

IPC: G06N3/08 ,  G06N3/006 ,  G06N3/084 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/006 ,  G06N3/045 ,  G06N3/084

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.

公开(公告)号：US20230084700A1

公开(公告)日：2023-03-16

申请号：US17948016

申请日：2022-09-19

Applicant: DeepMind Technologies Limited

Inventor： Karen Simonyan ,  David Silver ,  Julian Schrittwieser

IPC: G06N3/08 ,  G06N7/00

Abstract: Methods, systems and apparatus, including computer programs encoded on computer storage media, for training an action selection neural network. One of the methods includes receiving an observation characterizing a current state of the environment; determining a target network output for the observation by performing a look ahead search of possible future states of the environment starting from the current state until the environment reaches a possible future state that satisfies one or more termination criteria, wherein the look ahead search is guided by the neural network in accordance with current values of the network parameters; selecting an action to be performed by the agent in response to the observation using the target network output generated by performing the look ahead search; and storing, in an exploration history data store, the target network output in association with the observation for use in updating the current values of the network parameters.

公开(公告)号：US10650310B2

公开(公告)日：2020-05-12

申请号：US15349894

申请日：2016-11-11

Applicant: DeepMind Technologies Limited

Inventor： Tom Schaul ,  John Quan ,  David Silver

IPC: G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network used to select actions performed by a reinforcement learning agent interacting with an environment. In one aspect, a method includes maintaining a replay memory, where the replay memory stores pieces of experience data generated as a result of the reinforcement learning agent interacting with the environment. Each piece of experience data is associated with a respective expected learning progress measure that is a measure of an expected amount of progress made in the training of the neural network if the neural network is trained on the piece of experience data. The method further includes selecting a piece of experience data from the replay memory by prioritizing for selection pieces of experience data having relatively higher expected learning progress measures and training the neural network on the selected piece of experience data.

公开(公告)号：US20190354859A1

公开(公告)日：2019-11-21

申请号：US16417536

申请日：2019-05-20

Applicant: DeepMind Technologies Limited

Inventor： Zhongwen Xu ,  Hado Philip van Hasselt ,  David Silver

IPC: G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for reinforcement learning. The embodiments described herein apply meta-learning (and in particular, meta-gradient reinforcement learning) to learn an optimum return function G so that the training of the system is improved. This provides a more effective and efficient means of training a reinforcement learning system as the system is able to converge on an optimum set of one or more policy parameters θ more quickly by training the return function G as it goes. In particular, the return function G is made dependent on the one or more policy parameters θ and a meta-objective function J′ is used that is differentiated with respect to the one or more return parameters η to improve the training of the return function G.

公开(公告)号：US20180260708A1

公开(公告)日：2018-09-13

申请号：US15977923

申请日：2018-05-11

Applicant: DeepMind Technologies Limited

Inventor： Volodymyr Mnih ,  Adrià Puigdomènech Badia ,  Alexander Benjamin Graves ,  Timothy James Alexander Harley ,  David Silver ,  Koray Kavukcuoglu

IPC: G06N3/08 ,  G06N3/04

CPC classification number: G06N3/08 ,  G06N3/04 ,  G06N3/0454

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for asynchronous deep reinforcement learning. One of the systems includes a plurality of workers, wherein each worker is configured to operate independently of each other worker, and wherein each worker is associated with a respective actor that interacts with a respective replica of the environment during the training of the deep neural network.

公开(公告)号：US20250045583A1

公开(公告)日：2025-02-06

申请号：US18805367

申请日：2024-08-14

Applicant: DeepMind Technologies Limited

Inventor： Tom Schaul ,  John Quan ,  David Silver

IPC: G06N3/08 ,  G06N3/088

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network used to select actions performed by a reinforcement learning agent interacting with an environment. In one aspect, a method includes maintaining a replay memory, where the replay memory stores pieces of experience data generated as a result of the reinforcement learning agent interacting with the environment. Each piece of experience data is associated with a respective expected learning progress measure that is a measure of an expected amount of progress made in the training of the neural network if the neural network is trained on the piece of experience data. The method further includes selecting a piece of experience data from the replay memory by prioritizing for selection pieces of experience data having relatively higher expected learning progress measures and training the neural network on the selected piece of experience data.

公开(公告)号：US20240127045A1

公开(公告)日：2024-04-18

申请号：US17959210

申请日：2022-10-03

Applicant: DeepMind Technologies Limited

Inventor： Thomas Keisuke Hubert ,  Shih-Chieh Huang ,  Alexander Novikov ,  Alhussein Fawzi ,  Bernardino Romera-Paredes ,  David Silver ,  Demis Hassabis ,  Grzegorz Michal Swirszcz ,  Julian Schrittwieser ,  Pushmeet Kohli ,  Mohammadamin Barekatain ,  Matej Balog ,  Francisco Jesus Rodriguez Ruiz

IPC: G06N3/08 ,  G06N3/063

CPC classification number: G06N3/08 ,  G06N3/063

Abstract: A method performed by one or more computers for obtaining an optimized algorithm that (i) is functionally equivalent to a target algorithm and (ii) optimizes one or more target properties when executed on a target set of one or more hardware devices. The method includes: initializing a target tensor representing the target algorithm; generating, using a neural network having a plurality of network parameters, a tensor decomposition of the target tensor that parametrizes a candidate algorithm; generating target property values for each of the target properties when executing the candidate algorithm on the target set of hardware devices; determining a benchmarking score for the tensor decomposition based on the target property values of the candidate algorithm; generating a training example from the tensor decomposition and the benchmarking score; and storing, in a training data store, the training example for use in updating the network parameters of the neural network.