公开(公告)号：US12008077B1

公开(公告)日：2024-06-11

申请号：US18120912

申请日：2023-03-13

Applicant: DeepMind Technologies Limited

Inventor： Todd Andrew Hester

IPC: G06N3/08 ,  G06F17/11 ,  G06F18/214 ,  G06N3/045

CPC classification number: G06F18/2148 ,  G06F17/11 ,  G06N3/045 ,  G06N3/08

Abstract: A method of training an action selection neural network to perform a demonstrated task using a supervised learning technique. The action selection neural network is configured to receive demonstration data comprising actions to perform the task and rewards received for performing the actions. The action selection neural network has auxiliary prediction task neural networks on one or more of its intermediate outputs. The action selection policy neural network is trained using multiple combined losses, concurrently with the auxiliary prediction task neural networks.

公开(公告)号：US11886997B2

公开(公告)日：2024-01-30

申请号：US17962008

申请日：2022-10-07

Applicant: DeepMind Technologies Limited

Inventor： Olivier Pietquin ,  Martin Riedmiller ,  Wang Fumin ,  Bilal Piot ,  Mel Vecerik ,  Todd Andrew Hester ,  Thomas Rothoerl ,  Thomas Lampe ,  Nicolas Manfred Otto Heess ,  Jonathan Karl Scholz

IPC: G06N3/08 ,  G06N3/045 ,  G06N3/047

CPC classification number: G06N3/08 ,  G06N3/045 ,  G06N3/047

Abstract: An off-policy reinforcement learning actor-critic neural network system configured to select actions from a continuous action space to be performed by an agent interacting with an environment to perform a task. An observation defines environment state data and reward data. The system has an actor neural network which learns a policy function mapping the state data to action data. A critic neural network learns an action-value (Q) function. A replay buffer stores tuples of the state data, the action data, the reward data and new state data. The replay buffer also includes demonstration transition data comprising a set of the tuples from a demonstration of the task within the environment. The neural network system is configured to train the actor neural network and the critic neural network off-policy using stored tuples from the replay buffer comprising tuples both from operation of the system and from the demonstration transition data.

公开(公告)号：US20210287072A1

公开(公告)日：2021-09-16

申请号：US17331614

申请日：2021-05-26

Applicant: DeepMind Technologies Limited

Inventor： Richard Andrew Evans ,  Jim Gao ,  Michael C. Ryan ,  Gabriel Dulac-Arnold ,  Jonathan Karl Scholz ,  Todd Andrew Hester

IPC: G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for improving operational efficiency within a data center by modeling data center performance and predicting power usage efficiency. An example method receives a state input characterizing a current state of a data center. For each data center setting slate, the state input and the data center setting slate are processed through an ensemble of machine learning models. Each machine learning model is configured to receive and process the state input and the data center setting slate to generate an efficiency score that characterizes a predicted resource efficiency of the data center if the data center settings defined by the data center setting slate are adopted t. The method selects, based on the efficiency scores for the data center setting slates, new values for the data center settings.

公开(公告)号：US11868882B2

公开(公告)日：2024-01-09

申请号：US16624245

申请日：2018-06-28

Applicant: DEEPMIND TECHNOLOGIES LIMITED

Inventor： Olivier Claude Pietquin ,  Martin Riedmiller ,  Wang Fumin ,  Bilal Piot ,  Mel Vecerik ,  Todd Andrew Hester ,  Thomas Rothoerl ,  Thomas Lampe ,  Nicolas Manfred Otto Heess ,  Jonathan Karl Scholz

IPC: G06N3/02 ,  G06N3/08 ,  G06N3/045 ,  G06N3/047

CPC classification number: G06N3/08 ,  G06N3/045 ,  G06N3/047

Abstract: An off-policy reinforcement learning actor-critic neural network system configured to select actions from a continuous action space to be performed by an agent interacting with an environment to perform a task. An observation defines environment state data and reward data. The system has an actor neural network which learns a policy function mapping the state data to action data. A critic neural network learns an action-value (Q) function. A replay buffer stores tuples of the state data, the action data, the reward data and new state data. The replay buffer also includes demonstration transition data comprising a set of the tuples from a demonstration of the task within the environment. The neural network system is configured to train the actor neural network and the critic neural network off-policy using stored tuples from the replay buffer comprising tuples both from operation of the system and from the demonstration transition data.

公开(公告)号：US11836599B2

公开(公告)日：2023-12-05

申请号：US17331614

申请日：2021-05-26

Applicant: DeepMind Technologies Limited

Inventor： Richard Andrew Evans ,  Jim Gao ,  Michael C. Ryan ,  Gabriel Dulac-Arnold ,  Jonathan Karl Scholz ,  Todd Andrew Hester

IPC: G06N3/04 ,  G06N3/045

CPC classification number: G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for improving operational efficiency within a data center by modeling data center performance and predicting power usage efficiency. An example method receives a state input characterizing a current state of a data center. For each data center setting slate, the state input and the data center setting slate are processed through an ensemble of machine learning models. Each machine learning model is configured to receive and process the state input and the data center setting slate to generate an efficiency score that characterizes a predicted resource efficiency of the data center if the data center settings defined by the data center setting slate are adopted t. The method selects, based on the efficiency scores for the data center setting slates, new values for the data center settings.

公开(公告)号：US20220343157A1

公开(公告)日：2022-10-27

申请号：US17620164

申请日：2020-06-17

Applicant: DEEPMIND TECHNOLOGIES LIMITED

Inventor： Daniel J. Mankowitz ,  Nir Levine ,  Rae Chan Jeong ,  Abbas Abdolmaleki ,  Jost Tobias Springenberg ,  Todd Andrew Hester ,  Timothy Arthur Mann ,  Martin Riedmiller

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a policy neural network having policy parameters. One of the methods includes sampling a mini-batch comprising one or more observation-action-reward tuples generated as a result of interactions of a first agent with a first environment; determining an update to current values of the Q network parameters by minimizing a robust entropy-regularized temporal difference (TD) error that accounts for possible perturbations of the states of the first environment represented by the observations in the observation-action-reward tuples; and determining, using the Q-value neural network, an update to the policy network parameters using the sampled mini-batch of observation-action-reward tuples.

公开(公告)号：US20230023189A1

公开(公告)日：2023-01-26

申请号：US17962008

申请日：2022-10-07

Applicant: DeepMind Technologies Limited

Inventor： Olivier Pietquin ,  Martin Riedmiller ,  Wang Fumin ,  Bilal Piot ,  Mel Vecerik ,  Todd Andrew Hester ,  Thomas Rothoerl ,  Thomas Lampe ,  Nicolas Manfred Otto Heess ,  Jonathan Karl Scholz

IPC: G06N3/08 ,  G06N3/04

Abstract: An off-policy reinforcement learning actor-critic neural network system configured to select actions from a continuous action space to be performed by an agent interacting with an environment to perform a task. An observation defines environment state data and reward data. The system has an actor neural network which learns a policy function mapping the state data to action data. A critic neural network learns an action-value (Q) function. A replay buffer stores tuples of the state data, the action data, the reward data and new state data. The replay buffer also includes demonstration transition data comprising a set of the tuples from a demonstration of the task within the environment. The neural network system is configured to train the actor neural network and the critic neural network off-policy using stored tuples from the replay buffer comprising tuples both from operation of the system and from the demonstration transition data.

公开(公告)号：US11604941B1

公开(公告)日：2023-03-14

申请号：US16174148

申请日：2018-10-29

Applicant: DeepMind Technologies Limited

Inventor： Todd Andrew Hester

IPC: G06N3/08 ,  G06K9/62 ,  G06F17/11 ,  G06N3/04

Abstract: A method of training an action selection neural network to perform a demonstrated task using a supervised learning technique. The action selection neural network is configured to receive demonstration data comprising actions to perform the task and rewards received for performing the actions. The action selection neural network has auxiliary prediction task neural networks on one or more of its intermediate outputs. The action selection policy neural network is trained using multiple combined losses, concurrently with the auxiliary prediction task neural networks.

公开(公告)号：US11468321B2

公开(公告)日：2022-10-11

申请号：US16624245

申请日：2018-06-28

Applicant: DEEPMIND TECHNOLOGIES LIMITED

Inventor： Olivier Claude Pietquin ,  Martin Riedmiller ,  Wang Fumin ,  Bilal Piot ,  Mel Vecerik ,  Todd Andrew Hester ,  Thomas Rothoerl ,  Thomas Lampe ,  Nicolas Manfred Otto Heess ,  Jonathan Karl Scholz

IPC: G06N3/02 ,  G06N3/08 ,  G06N3/04

Abstract: An off-policy reinforcement learning actor-critic neural network system configured to select actions from a continuous action space to be performed by an agent interacting with an environment to perform a task. An observation defines environment state data and reward data. The system has an actor neural network which learns a policy function mapping the state data to action data. A critic neural network learns an action-value (Q) function. A replay buffer stores tuples of the state data, the action data, the reward data and new state data. The replay buffer also includes demonstration transition data comprising a set of the tuples from a demonstration of the task within the environment. The neural network system is configured to train the actor neural network and the critic neural network off-policy using stored tuples from the replay buffer comprising tuples both from operation of the system and from the demonstration transition data.

公开(公告)号：US20200272889A1

公开(公告)日：2020-08-27

申请号：US16863357

申请日：2020-04-30

Applicant: DeepMind Technologies Limited

Inventor： Richard Andrew Evans ,  Jim Gao ,  Michael C. Ryan ,  Gabriel Dulac-Arnold ,  Jonathan Karl Scholz ,  Todd Andrew Hester

IPC: G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for improving operational efficiency within a data center by modeling data center performance and predicting power usage efficiency. An example method receives a state input characterizing a current state of a data center. For each data center setting slate, the state input and the data center setting slate are processed through an ensemble of machine learning models. Each machine learning model is configured to receive and process the state input and the data center setting slate to generate an efficiency score that characterizes a predicted resource efficiency of the data center if the data center settings defined by the data center setting slate are adopted t. The method selects, based on the efficiency scores for the data center setting slates, new values for the data center settings.