公开(公告)号：US20240062035A1

公开(公告)日：2024-02-22

申请号：US18351440

申请日：2023-07-12

Applicant: DeepMind Technologies Limited

Inventor： Martin Riedmiller ,  Roland Hafner ,  Mel Vecerik ,  Timothy Paul Lillicrap ,  Thomas Lampe ,  Ivaylo Popov ,  Gabriel Barth-Maron ,  Nicolas Manfred Otto Heess

IPC: G06N3/006 ,  G06N3/08 ,  G06N3/088 ,  G06F18/21 ,  G06F18/214 ,  G06N3/045

CPC classification number: G06N3/006 ,  G06N3/08 ,  G06N3/088 ,  G06F18/2185 ,  G06F18/2148 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data-efficient reinforcement learning. One of the systems is a system for training an actor neural network used to select actions to be performed by an agent that interacts with an environment by receiving observations characterizing states of the environment and, in response to each observation, performing an action selected from a continuous space of possible actions, wherein the actor neural network maps observations to next actions in accordance with values of parameters of the actor neural network, and wherein the system comprises: a plurality of workers, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a respective agent replica that interacts with a respective replica of the environment during the training of the actor neural network.

公开(公告)号：US11868882B2

公开(公告)日：2024-01-09

申请号：US16624245

申请日：2018-06-28

Applicant: DEEPMIND TECHNOLOGIES LIMITED

Inventor： Olivier Claude Pietquin ,  Martin Riedmiller ,  Wang Fumin ,  Bilal Piot ,  Mel Vecerik ,  Todd Andrew Hester ,  Thomas Rothoerl ,  Thomas Lampe ,  Nicolas Manfred Otto Heess ,  Jonathan Karl Scholz

IPC: G06N3/02 ,  G06N3/08 ,  G06N3/045 ,  G06N3/047

CPC classification number: G06N3/08 ,  G06N3/045 ,  G06N3/047

Abstract: An off-policy reinforcement learning actor-critic neural network system configured to select actions from a continuous action space to be performed by an agent interacting with an environment to perform a task. An observation defines environment state data and reward data. The system has an actor neural network which learns a policy function mapping the state data to action data. A critic neural network learns an action-value (Q) function. A replay buffer stores tuples of the state data, the action data, the reward data and new state data. The replay buffer also includes demonstration transition data comprising a set of the tuples from a demonstration of the task within the environment. The neural network system is configured to train the actor neural network and the critic neural network off-policy using stored tuples from the replay buffer comprising tuples both from operation of the system and from the demonstration transition data.

公开(公告)号：US20200285909A1

公开(公告)日：2020-09-10

申请号：US16882373

申请日：2020-05-22

Applicant: DeepMind Technologies Limited

Inventor： Martin Riedmiller ,  Roland Hafner ,  Mel Vecerik ,  Timothy Paul Lillicrap ,  Thomas Lampe ,  Ivaylo Popov ,  Gabriel Barth-Maron ,  Nicolas Manfred Otto Heess

IPC: G06K9/62 ,  G06N3/04 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data-efficient reinforcement learning. One of the systems is a system for training an actor neural network used to select actions to be performed by an agent that interacts with an environment by receiving observations characterizing states of the environment and, in response to each observation, performing an action selected from a continuous space of possible actions, wherein the actor neural network maps observations to next actions in accordance with values of parameters of the actor neural network, and wherein the system comprises: a plurality of workers, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a respective agent replica that interacts with a respective replica of the environment during the training of the actor neural network.

公开(公告)号：US10664725B2

公开(公告)日：2020-05-26

申请号：US16528260

申请日：2019-07-31

Applicant: DeepMind Technologies Limited

Inventor： Martin Riedmiller ,  Roland Hafner ,  Mel Vecerik ,  Timothy Paul Lillicrap ,  Thomas Lampe ,  Ivaylo Popov ,  Gabriel Barth-Maron ,  Nicolas Manfred Otto Heess

IPC: G06N3/08 ,  G06K9/62 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data-efficient reinforcement learning. One of the systems is a system for training an actor neural network used to select actions to be performed by an agent that interacts with an environment by receiving observations characterizing states of the environment and, in response to each observation, performing an action selected from a continuous space of possible actions, wherein the actor neural network maps observations to next actions in accordance with values of parameters of the actor neural network, and wherein the system comprises: a plurality of workers, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a respective agent replica that interacts with a respective replica of the environment during the training of the actor neural network.

公开(公告)号：US20230023189A1

公开(公告)日：2023-01-26

申请号：US17962008

申请日：2022-10-07

Applicant: DeepMind Technologies Limited

Inventor： Olivier Pietquin ,  Martin Riedmiller ,  Wang Fumin ,  Bilal Piot ,  Mel Vecerik ,  Todd Andrew Hester ,  Thomas Rothoerl ,  Thomas Lampe ,  Nicolas Manfred Otto Heess ,  Jonathan Karl Scholz

IPC: G06N3/08 ,  G06N3/04

Abstract: An off-policy reinforcement learning actor-critic neural network system configured to select actions from a continuous action space to be performed by an agent interacting with an environment to perform a task. An observation defines environment state data and reward data. The system has an actor neural network which learns a policy function mapping the state data to action data. A critic neural network learns an action-value (Q) function. A replay buffer stores tuples of the state data, the action data, the reward data and new state data. The replay buffer also includes demonstration transition data comprising a set of the tuples from a demonstration of the task within the environment. The neural network system is configured to train the actor neural network and the critic neural network off-policy using stored tuples from the replay buffer comprising tuples both from operation of the system and from the demonstration transition data.

公开(公告)号：US11741334B2

公开(公告)日：2023-08-29

申请号：US16882373

申请日：2020-05-22

Applicant: DeepMind Technologies Limited

Inventor： Martin Riedmiller ,  Roland Hafner ,  Mel Vecerik ,  Timothy Paul Lillicrap ,  Thomas Lampe ,  Ivaylo Popov ,  Gabriel Barth-Maron ,  Nicolas Manfred Otto Heess

IPC: G06N3/08 ,  G06N3/006 ,  G06N3/088 ,  G06F18/21 ,  G06F18/214 ,  G06N3/045

CPC classification number: G06N3/006 ,  G06F18/2148 ,  G06F18/2185 ,  G06N3/045 ,  G06N3/08 ,  G06N3/088

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data-efficient reinforcement learning. One of the systems is a system for training an actor neural network used to select actions to be performed by an agent that interacts with an environment by receiving observations characterizing states of the environment and, in response to each observation, performing an action selected from a continuous space of possible actions, wherein the actor neural network maps observations to next actions in accordance with values of parameters of the actor neural network, and wherein the system comprises: a plurality of workers, wherein each worker is configured to operate independently of each other worker, wherein each worker is associated with a respective agent replica that interacts with a respective replica of the environment during the training of the actor neural network.

公开(公告)号：US11468321B2

公开(公告)日：2022-10-11

申请号：US16624245

申请日：2018-06-28

Applicant: DEEPMIND TECHNOLOGIES LIMITED

Inventor： Olivier Claude Pietquin ,  Martin Riedmiller ,  Wang Fumin ,  Bilal Piot ,  Mel Vecerik ,  Todd Andrew Hester ,  Thomas Rothoerl ,  Thomas Lampe ,  Nicolas Manfred Otto Heess ,  Jonathan Karl Scholz

IPC: G06N3/02 ,  G06N3/08 ,  G06N3/04

Abstract: An off-policy reinforcement learning actor-critic neural network system configured to select actions from a continuous action space to be performed by an agent interacting with an environment to perform a task. An observation defines environment state data and reward data. The system has an actor neural network which learns a policy function mapping the state data to action data. A critic neural network learns an action-value (Q) function. A replay buffer stores tuples of the state data, the action data, the reward data and new state data. The replay buffer also includes demonstration transition data comprising a set of the tuples from a demonstration of the task within the environment. The neural network system is configured to train the actor neural network and the critic neural network off-policy using stored tuples from the replay buffer comprising tuples both from operation of the system and from the demonstration transition data.

公开(公告)号：US20200151562A1

公开(公告)日：2020-05-14

申请号：US16624245

申请日：2018-06-28

Applicant: DEEPMIND TECHNOLOGIES LIMITED

Inventor： Olivier Pietquin ,  Martin Riedmiller ,  Wang Fumin ,  Bilal Piot ,  Mel Vecerik ,  Todd Andrew Hester ,  Thomas Rothörl ,  Thomas Lampe ,  Nicolas Manfred Otto Heess ,  Jonathan Karl Scholz

IPC: G06N3/08 ,  G06N3/04

Abstract: An off-policy reinforcement learning actor-critic neural network system configured to select actions from a continuous action space to be performed by an agent interacting with an environment to perform a task. An observation defines environment state data and reward data. The system has an actor neural network which learns a policy function mapping the state data to action data. A critic neural network learns an action-value (Q) function. A replay buffer stores tuples of the state data, the action data, the reward data and new state data. The replay buffer also includes demonstration transition data comprising a set of the tuples from a demonstration of the task within the environment. The neural network system is configured to train the actor neural network and the critic neural network off-policy using stored tuples from the replay buffer comprising tuples both from operation of the system and from the demonstration transition data.

公开(公告)号：US11886997B2

公开(公告)日：2024-01-30

申请号：US17962008

申请日：2022-10-07

Applicant: DeepMind Technologies Limited

Inventor： Olivier Pietquin ,  Martin Riedmiller ,  Wang Fumin ,  Bilal Piot ,  Mel Vecerik ,  Todd Andrew Hester ,  Thomas Rothoerl ,  Thomas Lampe ,  Nicolas Manfred Otto Heess ,  Jonathan Karl Scholz

IPC: G06N3/08 ,  G06N3/045 ,  G06N3/047

CPC classification number: G06N3/08 ,  G06N3/045 ,  G06N3/047

Abstract: An off-policy reinforcement learning actor-critic neural network system configured to select actions from a continuous action space to be performed by an agent interacting with an environment to perform a task. An observation defines environment state data and reward data. The system has an actor neural network which learns a policy function mapping the state data to action data. A critic neural network learns an action-value (Q) function. A replay buffer stores tuples of the state data, the action data, the reward data and new state data. The replay buffer also includes demonstration transition data comprising a set of the tuples from a demonstration of the task within the environment. The neural network system is configured to train the actor neural network and the critic neural network off-policy using stored tuples from the replay buffer comprising tuples both from operation of the system and from the demonstration transition data.

公开(公告)号：US20210103815A1

公开(公告)日：2021-04-08

申请号：US17065489

申请日：2020-10-07

Applicant: DeepMind Technologies Limited

Inventor： Rae Chan Jeong ,  Yusuf Aytar ,  David Khosid ,  Yuxiang Zhou ,  Jacqueline Ok-chan Kay ,  Thomas Lampe ,  Konstantinos Bousmalis ,  Francesco Nori

IPC: G06N3/08 ,  G05B19/4155

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a policy neural network for use in controlling a real-world agent in a real-world environment. One of the methods includes training the policy neural network by optimizing a first task-specific objective that measures a performance of the policy neural network in controlling a simulated version of the real-world agent; and then training the policy neural network by jointly optimizing (i) a self-supervised objective that measures at least a performance of internal representations generated by the policy neural network on a self-supervised task performed on real-world data and (ii) a second task-specific objective that measures the performance of the policy neural network in controlling the simulated version of the real-world agent.