公开(公告)号：US20220261639A1

公开(公告)日：2022-08-18

申请号：US17625361

申请日：2020-07-16

Applicant: DeepMind Technologies Limited

Inventor： Konrad Zolna ,  Scott Ellison Reed ,  Ziyu Wang ,  Alexander Novikov ,  Sergio Gomez Colmenarejo ,  Joao Ferdinando Gomes de Freitas ,  David Budden ,  Serkan Cabi

IPC: G06N3/08

Abstract: A method is proposed of training a neural network to generate action data for controlling an agent to perform a task in an environment. The method includes obtaining, for each of a plurality of performances of the task, one or more first tuple datasets, each first tuple dataset comprising state data characterizing a state of the environment at a corresponding time during the performance of the task; and a concurrent process of training the neural network and a discriminator network. The training process comprises a plurality of neural network update steps and a plurality of discriminator network update steps. Each neural network update step comprises: receiving state data characterizing a current state of the environment; using the neural network and the state data to generate action data indicative of an action to be performed by the agent; forming a second tuple dataset comprising the state data; using the second tuple dataset to generate a reward value, wherein the reward value comprises an imitation value generated by the discriminator network based on the second tuple dataset; and updating one or more parameters of the neural network based on the reward value. Each discriminator network update step comprises updating the discriminator network based on a plurality of the first tuple datasets and a plurality of the second tuple datasets, the update being to increase respective imitation values which the discriminator network generates upon receiving any of the plurality of the first tuple datasets compared to respective imitation values which the discriminator network generates upon receiving any of the plurality of the second tuple datasets. The updating process is performed subject to a constraint that the updated discriminator network, upon receiving any of at least a certain proportion of a first subset of the first tuple datasets and/or any of at least a certain proportion of a second subset of the second tuple datasets, does not generate imitation values which correctly indicate that those tuple datasets are first or second tuple datasets.

公开(公告)号：US12277497B2

公开(公告)日：2025-04-15

申请号：US18131753

申请日：2023-04-06

Applicant: DeepMind Technologies Limited

Inventor： David Budden ,  Gabriel Barth-Maron ,  John Quan ,  Daniel George Horgan

IPC: G06N3/08 ,  G06N3/04 ,  G06N3/088 ,  G06N20/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an action selection neural network used to select actions to be performed by an agent interacting with an environment. One of the systems includes (i) a plurality of actor computing units, in which each of the actor computing units is configured to maintain a respective replica of the action selection neural network and to perform a plurality of actor operations, and (ii) one or more learner computing units, in which each of the one or more learner computing units is configured to perform a plurality of learner operations.

公开(公告)号：US20240042600A1

公开(公告)日：2024-02-08

申请号：US18331632

申请日：2023-06-08

Applicant: DeepMind Technologies Limited

Inventor： Serkan Cabi ,  Ziyu Wang ,  Alexander Novikov ,  Ksenia Konyushkova ,  Sergio Gomez Colmenarejo ,  Scott Ellison Reed ,  Misha Man Ray Denil ,  Jonathan Karl Scholz ,  Oleg O. Sushkov ,  Rae Chan Jeong ,  David Barker ,  David Budden ,  Mel Vecerik ,  Yusuf Aytar ,  Joao Ferdinando Gomes de Freitas

IPC: B25J9/16

CPC classification number: B25J9/161 ,  B25J9/163 ,  B25J9/1661

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data-driven robotic control. One of the methods includes maintaining robot experience data; obtaining annotation data; training, on the annotation data, a reward model; generating task-specific training data for the particular task, comprising, for each experience in a second subset of the experiences in the robot experience data: processing the observation in the experience using the trained reward model to generate a reward prediction, and associating the reward prediction with the experience; and training a policy neural network on the task-specific training data for the particular task, wherein the policy neural network is configured to receive a network input comprising an observation and to generate a policy output that defines a control policy for a robot performing the particular task.

公开(公告)号：US11712799B2

公开(公告)日：2023-08-01

申请号：US17020294

申请日：2020-09-14

Applicant: DeepMind Technologies Limited

Inventor： Serkan Cabi ,  Ziyu Wang ,  Alexander Novikov ,  Ksenia Konyushkova ,  Sergio Gomez Colmenarejo ,  Scott Ellison Reed ,  Misha Man Ray Denil ,  Jonathan Karl Scholz ,  Oleg O. Sushkov ,  Rae Chan Jeong ,  David Barker ,  David Budden ,  Mel Vecerik ,  Yusuf Aytar ,  Joao Ferdinando Gomes de Freitas

IPC: B25J9/16

CPC classification number: B25J9/161 ,  B25J9/163 ,  B25J9/1661

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data-driven robotic control. One of the methods includes maintaining robot experience data; obtaining annotation data; training, on the annotation data, a reward model; generating task-specific training data for the particular task, comprising, for each experience in a second subset of the experiences in the robot experience data: processing the observation in the experience using the trained reward model to generate a reward prediction, and associating the reward prediction with the experience; and training a policy neural network on the task-specific training data for the particular task, wherein the policy neural network is configured to receive a network input comprising an observation and to generate a policy output that defines a control policy for a robot performing the particular task.

公开(公告)号：US11663475B2

公开(公告)日：2023-05-30

申请号：US17945622

申请日：2022-09-15

Applicant: DeepMind Technologies Limited

Inventor： David Budden ,  Matthew William Hoffman ,  Gabriel Barth-Maron

IPC: G06N3/08 ,  G06N3/045 ,  G06N3/04

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an action selection neural network that is used to select actions to be performed by a reinforcement learning agent interacting with an environment. In particular, the actions are selected from a continuous action space and the system trains the action selection neural network jointly with a distribution Q network that is used to update the parameters of the action selection neural network.

公开(公告)号：US11663441B2

公开(公告)日：2023-05-30

申请号：US16586437

申请日：2019-09-27

Applicant: DeepMind Technologies Limited

Inventor： Scott Ellison Reed ,  Yusuf Aytar ,  Ziyu Wang ,  Tom Paine ,  Sergio Gomez Colmenarejo ,  David Budden ,  Tobias Pfaff ,  Aaron Gerard Antonius van den Oord ,  Oriol Vinyals ,  Alexander Novikov

IPC: G06N3/006 ,  G06F17/16 ,  G06N3/08 ,  G06F18/22 ,  G06N3/045 ,  G06N3/048 ,  G06V10/764 ,  G06V10/77 ,  G06V10/82

CPC classification number: G06N3/006 ,  G06F17/16 ,  G06F18/22 ,  G06N3/045 ,  G06N3/048 ,  G06N3/08 ,  G06V10/764 ,  G06V10/7715 ,  G06V10/82

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an action selection policy neural network, wherein the action selection policy neural network is configured to process an observation characterizing a state of an environment to generate an action selection policy output, wherein the action selection policy output is used to select an action to be performed by an agent interacting with an environment. In one aspect, a method comprises: obtaining an observation characterizing a state of the environment subsequent to the agent performing a selected action; generating a latent representation of the observation; processing the latent representation of the observation using a discriminator neural network to generate an imitation score; determining a reward from the imitation score; and adjusting the current values of the action selection policy neural network parameters based on the reward using a reinforcement learning training technique.

公开(公告)号：US20210078169A1

公开(公告)日：2021-03-18

申请号：US17020294

申请日：2020-09-14

Applicant: DeepMind Technologies Limited

Inventor： Serkan Cabi ,  Ziyu Wang ,  Alexander Novikov ,  Ksenia Konyushkova ,  Sergio Gomez Colmenarejo ,  Scott Ellison Reed ,  Misha Man Ray Denil ,  Jonathan Karl Scholz ,  Oleg O. Sushkov ,  Rae Chan Jeong ,  David Barker ,  David Budden ,  Mel Vecerik ,  Yusuf Aytar ,  Joao Ferdinando Gomes de Freitas

IPC: B25J9/16

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data-driven robotic control. One of the methods includes maintaining robot experience data; obtaining annotation data; training, on the annotation data, a reward model; generating task-specific training data for the particular task, comprising, for each experience in a second subset of the experiences in the robot experience data: processing the observation in the experience using the trained reward model to generate a reward prediction, and associating the reward prediction with the experience; and training a policy neural network on the task-specific training data for the particular task, wherein the policy neural network is configured to receive a network input comprising an observation and to generate a policy output that defines a control policy for a robot performing the particular task.

公开(公告)号：US20200293883A1

公开(公告)日：2020-09-17

申请号：US16759519

申请日：2018-10-29

Applicant: DeepMind Technologies Limited

Inventor： David Budden ,  Matthew William Hoffman ,  Gabriel Barth-Maron

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an action selection neural network that is used to select actions to be performed by a reinforcement learning agent interacting with an environment. In particular, the actions are selected from a continuous action space and the system trains the action selection neural network jointly with a distribution Q network that is used to update the parameters of the action selection neural network.