公开(公告)号：US20240062035A1

公开(公告)日：2024-02-22

申请号：US18351440

申请日：2023-07-12

Applicant: DeepMind Technologies Limited

Inventor： Martin Riedmiller ,  Roland Hafner ,  Mel Vecerik ,  Timothy Paul Lillicrap ,  Thomas Lampe ,  Ivaylo Popov ,  Gabriel Barth-Maron ,  Nicolas Manfred Otto Heess

IPC: G06N3/006 ,  G06N3/08 ,  G06N3/088 ,  G06F18/21 ,  G06F18/214 ,  G06N3/045

CPC classification number: G06N3/006 ,  G06N3/08 ,  G06N3/088 ,  G06F18/2185 ,  G06F18/2148 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data-efficient reinforcement learning. One of the systems is a system for training an actor neural network used to select actions to be performed by an agent that interacts with an environment by receiving observations characterizing states of the environment and, in response to each observation, performing an action selected from a continuous space of possible actions, wherein the actor neural network maps observations to next actions in accordance with values of parameters of the actor neural network, and wherein the system comprises: a plurality of workers, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a respective agent replica that interacts with a respective replica of the environment during the training of the actor neural network.

公开(公告)号：US11868882B2

公开(公告)日：2024-01-09

申请号：US16624245

申请日：2018-06-28

Applicant: DEEPMIND TECHNOLOGIES LIMITED

Inventor： Olivier Claude Pietquin ,  Martin Riedmiller ,  Wang Fumin ,  Bilal Piot ,  Mel Vecerik ,  Todd Andrew Hester ,  Thomas Rothoerl ,  Thomas Lampe ,  Nicolas Manfred Otto Heess ,  Jonathan Karl Scholz

IPC: G06N3/02 ,  G06N3/08 ,  G06N3/045 ,  G06N3/047

CPC classification number: G06N3/08 ,  G06N3/045 ,  G06N3/047

Abstract: An off-policy reinforcement learning actor-critic neural network system configured to select actions from a continuous action space to be performed by an agent interacting with an environment to perform a task. An observation defines environment state data and reward data. The system has an actor neural network which learns a policy function mapping the state data to action data. A critic neural network learns an action-value (Q) function. A replay buffer stores tuples of the state data, the action data, the reward data and new state data. The replay buffer also includes demonstration transition data comprising a set of the tuples from a demonstration of the task within the environment. The neural network system is configured to train the actor neural network and the critic neural network off-policy using stored tuples from the replay buffer comprising tuples both from operation of the system and from the demonstration transition data.

公开(公告)号：US11803750B2

公开(公告)日：2023-10-31

申请号：US17019927

申请日：2020-09-14

Applicant: DeepMind Technologies Limited

Inventor： Timothy Paul Lillicrap ,  Jonathan James Hunt ,  Alexander Pritzel ,  Nicolas Manfred Otto Heess ,  Tom Erez ,  Yuval Tassa ,  David Silver ,  Daniel Pieter Wierstra

IPC: G06N3/08 ,  G06N3/006 ,  G06N3/084 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/006 ,  G06N3/045 ,  G06N3/084

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an actor neural network used to select actions to be performed by an agent interacting with an environment. One of the methods includes obtaining a minibatch of experience tuples; and updating current values of the parameters of the actor neural network, comprising: for each experience tuple in the minibatch: processing the training observation and the training action in the experience tuple using a critic neural network to determine a neural network output for the experience tuple, and determining a target neural network output for the experience tuple; updating current values of the parameters of the critic neural network using errors between the target neural network outputs and the neural network outputs; and updating the current values of the parameters of the actor neural network using the critic neural network.

公开(公告)号：US20220366245A1

公开(公告)日：2022-11-17

申请号：US17763901

申请日：2020-09-23

Applicant: DeepMind Technologies Limited

Inventor： Arthur Clement Guez ,  Fabio Viola ,  Theophane Guillaume Weber ,  Lars Buesing ,  Nicolas Manfred Otto Heess

IPC: G06N3/08

Abstract: A reinforcement learning method and system that selects actions to be performed by a reinforcement learning agent interacting with an environment. A causal model is implemented by a hindsight model neural network and trained using hindsight i.e. using future environment state trajectories. As the method and system does not have access to this future information when selecting an action, the hindsight model neural network is used to train a model neural network which is conditioned on data from current observations, which learns to predict an output of the hindsight model neural network.

公开(公告)号：US20210089834A1

公开(公告)日：2021-03-25

申请号：US17114324

申请日：2020-12-07

Applicant: DeepMind Technologies Limited

Inventor： Daniel Pieter Wierstra ,  Yujia Li ,  Razvan Pascanu ,  Peter William Battaglia ,  Theophane Guillaume Weber ,  Lars Buesing ,  David Paul Reichert ,  Oriol Vinyals ,  Nicolas Manfred Otto Heess ,  Sebastien Henri Andre Racaniere

IPC: G06K9/62 ,  G06K9/68 ,  G06N3/08 ,  G06N5/04

Abstract: A neural network system is proposed to select actions to be performed by an agent interacting with an environment to perform a task in an attempt to achieve a specified result. The system may include a controller to receive state data and context data, and to output action data. The system may also include an imagination module to receive the state and action data, and to output consequent state data. The system may also include a manager to receive the state data and the context data, and to output route data which defines whether the system is to execute an action or to imagine. The system may also include a memory to store the context data.

公开(公告)号：US20250094772A1

公开(公告)日：2025-03-20

申请号：US18962266

申请日：2024-11-27

Applicant: DeepMind Technologies Limited

Inventor： Ziyu Wang ,  Nicolas Manfred Otto Heess ,  Victor Constant Bapst

IPC: G06N3/045 ,  G06N3/006 ,  G06N3/047 ,  G06N3/084 ,  G06N3/088

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an action selection neural network. One of the methods includes maintaining a replay memory that stores trajectories generated as a result of interaction of an agent with an environment; and training an action selection neural network having policy parameters on the trajectories in the replay memory, wherein training the action selection neural network comprises: sampling a trajectory from the replay memory; and adjusting current values of the policy parameters by training the action selection neural network on the trajectory using an off-policy actor critic reinforcement learning technique.

公开(公告)号：US20240403652A1

公开(公告)日：2024-12-05

申请号：US18699012

申请日：2022-10-05

Applicant: DeepMind Technologies Limited

Inventor： Dushyant ` Rao ,  Fereshteh Sadeghi ,  Leonard Hasenclever ,  Markus Wulfmeier ,  Martina Zambelli ,  Giulia Vezzani ,  Dhruva Tirumala Bukkapatnam ,  Yusuf Aytar ,  Joshua Merel ,  Nicolas Manfred Otto Heess ,  Raia Thais Hadsell

IPC: G06N3/092

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling agents. In particular, an agent can be controlled using a hierarchical controller that includes a high-level controller neural network, a mid-level controller neural network, and a low-level controller neural network.

公开(公告)号：US20230023189A1

公开(公告)日：2023-01-26

申请号：US17962008

申请日：2022-10-07

Applicant: DeepMind Technologies Limited

Inventor： Olivier Pietquin ,  Martin Riedmiller ,  Wang Fumin ,  Bilal Piot ,  Mel Vecerik ,  Todd Andrew Hester ,  Thomas Rothoerl ,  Thomas Lampe ,  Nicolas Manfred Otto Heess ,  Jonathan Karl Scholz

IPC: G06N3/08 ,  G06N3/04

Abstract: An off-policy reinforcement learning actor-critic neural network system configured to select actions from a continuous action space to be performed by an agent interacting with an environment to perform a task. An observation defines environment state data and reward data. The system has an actor neural network which learns a policy function mapping the state data to action data. A critic neural network learns an action-value (Q) function. A replay buffer stores tuples of the state data, the action data, the reward data and new state data. The replay buffer also includes demonstration transition data comprising a set of the tuples from a demonstration of the task within the environment. The neural network system is configured to train the actor neural network and the critic neural network off-policy using stored tuples from the replay buffer comprising tuples both from operation of the system and from the demonstration transition data.

公开(公告)号：US20220355472A1

公开(公告)日：2022-11-10

申请号：US17872528

申请日：2022-07-25

Applicant: DeepMind Technologies Limited

Inventor： Razvan Pascanu ,  Raia Thais Hadsell ,  Mel Vecerik ,  Thomas Rothoerl ,  Andrei-Alexandru Rusu ,  Nicolas Manfred Otto Heess

IPC: B25J9/16 ,  G06N3/04 ,  G06N3/08 ,  G05B13/02 ,  G06N3/00

Abstract: A system includes a neural network system implemented by one or more computers. The neural network system is configured to receive an observation characterizing a current state of a real-world environment being interacted with by a robotic agent to perform a robotic task and to process the observation to generate a policy output that defines an action to be performed by the robotic agent in response to the observation. The neural network system includes: (i) a sequence of deep neural networks (DNNs), in which the sequence of DNNs includes a simulation-trained DNN that has been trained on interactions of a simulated version of the robotic agent with a simulated version of the real-world environment to perform a simulated version of the robotic task, and (ii) a first robot-trained DNN that is configured to receive the observation and to process the observation to generate the policy output.

公开(公告)号：US20210073594A1

公开(公告)日：2021-03-11

申请号：US17019919

申请日：2020-09-14

Applicant: DeepMind Technologies Limited

Inventor： Daniel Pieter Wierstra ,  Yujia Li ,  Razvan Pascanu ,  Peter William Battaglia ,  Theophane Guillaume Weber ,  Lars Buesing ,  David Paul Reichert ,  Arthur Clement Guez ,  Danilo Jimenez Rezende ,  Adrià Puigdomènech Badia ,  Oriol Vinyals ,  Nicolas Manfred Otto Heess ,  Sebastien Henri Andre Racaniere

IPC: G06K9/62 ,  G06K9/68 ,  G06N3/04 ,  G06N3/08

Abstract: A neural network system is proposed. The neural network can be trained by model-based reinforcement learning to select actions to be performed by an agent interacting with an environment, to perform a task in an attempt to achieve a specified result. The system may comprise at least one imagination core which receives a current observation characterizing a current state of the environment, and optionally historical observations, and which includes a model of the environment. The imagination core may be configured to output trajectory data in response to the current observation, and/or historical observations. The trajectory data comprising a sequence of future features of the environment imagined by the imagination core. The system may also include a rollout encoder to encode the features, and an output stage to receive data derived from the rollout embedding and to output action policy data for identifying an action based on the current observation.