公开(公告)号：US20230325635A1

公开(公告)日：2023-10-12

申请号：US18025304

申请日：2021-09-10

Applicant: DeepMind Technologies Limited

Inventor： David Constantine Patrick Warde-Farley ,  Steven Stenberg Hansen ,  Volodymyr Mnih ,  Kate Alexandra Baumli

IPC: G06N3/045 ,  G06N3/08

CPC classification number: G06N3/045 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a policy neural network for use in controlling an agent using relative variational intrinsic control. In one aspect, a method includes: selecting a skill from a set of skills; generating a trajectory by controlling the agent using the policy neural network while the policy neural network is conditioned on the selected skill; processing an initial observation and a last observation using a relative discriminator neural network to generate a relative score; processing the last observation using an absolute discriminator neural network to generate an absolute score; generating a reward for the trajectory from the absolute score corresponding to the selected skill and the relative score corresponding to the selected skill; and training the policy neural network on the reward for the trajectory.

公开(公告)号：US20210065012A1

公开(公告)日：2021-03-04

申请号：US17020248

申请日：2020-09-14

Applicant: DeepMind Technologies Limited

Inventor： Mohammad Gheshlaghi Azar ,  Meire Fortunato ,  Bilal Piot ,  Olivier Claude Pietquin ,  Jacob Lee Menick ,  Volodymyr Mnih ,  Charles Blundell ,  Remi Munos

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for selecting an action to be performed by a reinforcement learning agent. The method includes obtaining an observation characterizing a current state of an environment. For each layer parameter of each noisy layer of a neural network, a respective noise value is determined. For each layer parameter of each noisy layer, a noisy current value for the layer parameter is determined from a current value of the layer parameter, a current value of a corresponding noise parameter, and the noise value. A network input including the observation is processed using the neural network in accordance with the noisy current values to generate a network output for the network input. An action is selected from a set of possible actions to be performed by the agent in response to the observation using the network output.

公开(公告)号：US20180260708A1

公开(公告)日：2018-09-13

申请号：US15977923

申请日：2018-05-11

Applicant: DeepMind Technologies Limited

Inventor： Volodymyr Mnih ,  Adrià Puigdomènech Badia ,  Alexander Benjamin Graves ,  Timothy James Alexander Harley ,  David Silver ,  Koray Kavukcuoglu

IPC: G06N3/08 ,  G06N3/04

CPC classification number: G06N3/08 ,  G06N3/04 ,  G06N3/0454

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for asynchronous deep reinforcement learning. One of the systems includes a plurality of workers, wherein each worker is configured to operate independently of each other worker, and wherein each worker is associated with a respective actor that interacts with a respective replica of the environment during the training of the deep neural network.

公开(公告)号：US12020155B2

公开(公告)日：2024-06-25

申请号：US17733594

申请日：2022-04-29

Applicant: DeepMind Technologies Limited

Inventor： Volodymyr Mnih ,  Adrià Puigdomènech Badia ,  Alexander Benjamin Graves ,  Timothy James Alexander Harley ,  David Silver ,  Koray Kavukcuoglu

IPC: G06N3/08 ,  G06N3/04 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/04 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for asynchronous deep reinforcement learning. One of the systems includes a plurality of workers, wherein each worker is configured to operate independently of each other worker, and wherein each worker is associated with a respective actor that interacts with a respective replica of the environment during the training of the deep neural network.

公开(公告)号：US11593646B2

公开(公告)日：2023-02-28

申请号：US16767049

申请日：2019-02-05

Applicant: DeepMind Technologies Limited

Inventor： Hubert Josef Soyer ,  Lasse Espeholt ,  Karen Simonyan ,  Yotam Doron ,  Vlad Firoiu ,  Volodymyr Mnih ,  Koray Kavukcuoglu ,  Remi Munos ,  Thomas Ward ,  Timothy James Alexander Harley ,  Iain Robert Dunning

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an action selection neural network used to select actions to be performed by an agent interacting with an environment. In one aspect, a system comprises a plurality of actor computing units and a plurality of learner computing units. The actor computing units generate experience tuple trajectories that are used by the learner computing units to update learner action selection neural network parameters using a reinforcement learning technique. The reinforcement learning technique may be an off-policy actor critic reinforcement learning technique.

公开(公告)号：US11507827B2

公开(公告)日：2022-11-22

申请号：US16601455

申请日：2019-10-14

Applicant: DeepMind Technologies Limited

Inventor： Praveen Deepak Srinivasan ,  Rory Fearon ,  Cagdas Alcicek ,  Arun Sarath Nair ,  Samuel Blackwell ,  Vedavyas Panneershelvam ,  Alessandro De Maria ,  Volodymyr Mnih ,  Koray Kavukcuoglu ,  David Silver ,  Mustafa Suleyman

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributed training of reinforcement learning systems. One of the methods includes receiving, by a learner, current values of the parameters of the Q network from a parameter server, wherein each learner maintains a respective learner Q network replica and a respective target Q network replica; updating, by the learner, the parameters of the learner Q network replica maintained by the learner using the current values; selecting, by the learner, an experience tuple from a respective replay memory; computing, by the learner, a gradient from the experience tuple using the learner Q network replica maintained by the learner and the target Q network replica maintained by the learner; and providing, by the learner, the computed gradient to the parameter server.

公开(公告)号：US11049008B2

公开(公告)日：2021-06-29

申请号：US15619393

申请日：2017-06-09

Applicant: DeepMind Technologies Limited

Inventor： Volodymyr Mnih ,  Koray Kavukcuoglu

IPC: G06N3/08 ,  G06N20/00 ,  G06N3/04 ,  A63F13/67

Abstract: We describe a method of reinforcement learning for a subject system having multiple states and actions to move from one state to the next. Training data is generated by operating on the system with a succession of actions and used to train a second neural network. Target values for training the second neural network are derived from a first neural network which is generated by copying weights of the second neural network at intervals.

公开(公告)号：US10748041B1

公开(公告)日：2020-08-18

申请号：US16250320

申请日：2019-01-17

Applicant: DeepMind Technologies Limited

Inventor： Volodymyr Mnih ,  Koray Kavukcuoglu

IPC: G06K9/62 ,  G06K9/00 ,  G06K9/46 ,  G06K9/66

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing images using recurrent attention. One of the methods includes determining a location in the first image; extracting a glimpse from the first image using the location; generating a glimpse representation of the extracted glimpse; processing the glimpse representation using a recurrent neural network to update a current internal state of the recurrent neural network to generate a new internal state; processing the new internal state to select a location in a next image in the image sequence after the first image; and processing the new internal state to select an action from a predetermined set of possible actions.

公开(公告)号：US10445641B2

公开(公告)日：2019-10-15

申请号：US15016173

申请日：2016-02-04

Applicant: DeepMind Technologies Limited

Inventor： Praveen Deepak Srinivasan ,  Rory Fearon ,  Cagdas Alcicek ,  Arun Sarath Nair ,  Samuel Blackwell ,  Vedavyas Panneershelvam ,  Alessandro De Maria ,  Volodymyr Mnih ,  Koray Kavukcuoglu ,  David Silver ,  Mustafa Suleyman

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributed training of reinforcement learning systems. One of the methods includes receiving, by a learner, current values of the parameters of the Q network from a parameter server, wherein each learner maintains a respective learner Q network replica and a respective target Q network replica; updating, by the learner, the parameters of the learner Q network replica maintained by the learner using the current values; selecting, by the learner, an experience tuple from a respective replay memory; computing, by the learner, a gradient from the experience tuple using the learner Q network replica maintained by the learner and the target Q network replica maintained by the learner; and providing, by the learner, the computed gradient to the parameter server.

公开(公告)号：US20190258938A1

公开(公告)日：2019-08-22

申请号：US16403385

申请日：2019-05-03

Applicant: DeepMind Technologies Limited

Inventor： Volodymyr Mnih ,  Wojciech Czarnecki ,  Maxwell Elliot Jaderberg ,  Tom Schaul ,  David Silver ,  Koray Kavukcuoglu

IPC: G06N3/08 ,  G06N20/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a reinforcement learning system. The method includes: training an action selection policy neural network, and during the training of the action selection neural network, training one or more auxiliary control neural networks and a reward prediction neural network. Each of the auxiliary control neural networks is configured to receive a respective intermediate output generated by the action selection policy neural network and generate a policy output for a corresponding auxiliary control task. The reward prediction neural network is configured to receive one or more intermediate outputs generated by the action selection policy neural network and generate a corresponding predicted reward. Training each of the auxiliary control neural networks and the reward prediction neural network comprises adjusting values of the respective auxiliary control parameters, reward prediction parameters, and the action selection policy network parameters.