公开(公告)号：US12210943B2

公开(公告)日：2025-01-28

申请号：US17161845

申请日：2021-01-29

Applicant: GOOGLE LLC

Inventor： Adrian Li ,  Benjamin Holson ,  Alexander Herzog ,  Mrinal Kalakrishnan

IPC: G06N20/00 ,  G06N3/008 ,  G06N5/04

Abstract: Implementations disclosed herein relate to utilizing at least one existing manually engineered policy, for a robotic task, in training an RL policy model that can be used to at least selectively replace a portion of the engineered policy. The RL policy model can be trained for replacing a portion of a robotic task and can be trained based on data from episodes of attempting performance of the robotic task, including episodes in which the portion is performed based on the engineered policy and/or other portion(s) are performed based on the engineered policy. Once trained, the RL policy model can be used, at least selectively and in lieu of utilization of the engineered policy, to perform the portion of robotic task, while other portion(s) of the robotic task are performed utilizing the engineered policy and/or other similarly trained (but distinct) RL policy model(s).

公开(公告)号：US20220105624A1

公开(公告)日：2022-04-07

申请号：US17422260

申请日：2020-01-23

Applicant: Google LLC

Inventor： Mrinal Kalakrishnan ,  Yunfei Bai ,  Paul Wohlhart ,  Eric Jang ,  Chelsea Finn ,  Seyed Mohammad Khansari Zadeh ,  Sergey Levine ,  Allan Zhou ,  Alexander Herzog ,  Daniel Kappler

IPC: B25J9/16

Abstract: Techniques are disclosed that enable training a meta-learning model, for use in causing a robot to perform a task, using imitation learning as well as reinforcement learning. Some implementations relate to training the meta-learning model using imitation learning based on one or more human guided demonstrations of the task. Additional or alternative implementations relate to training the meta-learning model using reinforcement learning based on trials of the robot attempting to perform the task. Further implementations relate to using the trained meta-learning model to few shot (or one shot) learn a new task based on a human guided demonstration of the new task.

公开(公告)号：US20210237266A1

公开(公告)日：2021-08-05

申请号：US17052679

申请日：2019-06-14

Applicant: Google LLC

Inventor： Dmitry Kalashnikov ,  Alexander Irpan ,  Peter Pastor Sampedro ,  Julian Ibarz ,  Alexander Herzog ,  Eric Jang ,  Deirdre Quillen ,  Ethan Holly ,  Sergey Levine

IPC: B25J9/16 ,  G06N3/08

Abstract: Using large-scale reinforcement learning to train a policy model that can be utilized by a robot in performing a robotic task in which the robot interacts with one or more environmental objects. In various implementations, off-policy deep reinforcement learning is used to train the policy model, and the off-policy deep reinforcement learning is based on self-supervised data collection. The policy model can be a neural network model. Implementations of the reinforcement learning utilized in training the neural network model utilize a continuous-action variant of Q-learning. Through techniques disclosed herein, implementations can learn policies that generalize effectively to previously unseen objects, previously unseen environments, etc.

公开(公告)号：US20250131335A1

公开(公告)日：2025-04-24

申请号：US18991973

申请日：2024-12-23

Applicant: GOOGLE LLC

Inventor： Adrian Li ,  Benjamin Holson ,  Alexander Herzog ,  Mrinal Kalakrishnan

IPC: G06N20/00 ,  G06N3/008 ,  G06N5/04

Abstract: Implementations disclosed herein relate to utilizing at least one existing manually engineered policy, for a robotic task, in training an RL policy model that can be used to at least selectively replace a portion of the engineered policy. The RL policy model can be trained for replacing a portion of a robotic task and can be trained based on data from episodes of attempting performance of the robotic task, including episodes in which the portion is performed based on the engineered policy and/or other portion(s) are performed based on the engineered policy. Once trained, the RL policy model can be used, at least selectively and in lieu of utilization of the engineered policy, to perform the portion of robotic task, while other portion(s) of the robotic task are performed utilizing the engineered policy and/or other similarly trained (but distinct) RL policy model(s).

公开(公告)号：US11833661B2

公开(公告)日：2023-12-05

申请号：US17515490

申请日：2021-10-31

Applicant: GOOGLE LLC

Inventor： Zhuo Xu ,  Wenhao Yu ,  Alexander Herzog ,  Wenlong Lu ,  Chuyuan Fu ,  Yunfei Bai ,  C. Karen Liu ,  Daniel Ho

IPC: B25J9/16 ,  B25J13/08 ,  B25J19/02

CPC classification number: B25J9/163 ,  B25J9/161 ,  B25J9/1697 ,  B25J13/085 ,  B25J19/023

Abstract: Utilization of past dynamics sample(s), that reflect past contact physics information, in training and/or utilizing a neural network model. The neural network model represents a learned value function (e.g., a Q-value function) and that, when trained, can be used in selecting a sequence of robotic actions to implement in robotic manipulation (e.g., pushing) of an object by a robot. In various implementations, a past dynamics sample for an episode of robotic manipulation can include at least two past images from the episode, as well as one or more past force sensor readings that temporally correspond to the past images from the episode.

公开(公告)号：US12083678B2

公开(公告)日：2024-09-10

申请号：US17422260

申请日：2020-01-23

Applicant: Google LLC

Inventor： Mrinal Kalakrishnan ,  Yunfei Bai ,  Paul Wohlhart ,  Eric Jang ,  Chelsea Finn ,  Seyed Mohammad Khansari Zadeh ,  Sergey Levine ,  Allan Zhou ,  Alexander Herzog ,  Daniel Kappler

IPC: B25J9/16

CPC classification number: B25J9/163 ,  G05B2219/40116 ,  G05B2219/40499

Abstract: Techniques are disclosed that enable training a meta-learning model, for use in causing a robot to perform a task, using imitation learning as well as reinforcement learning. Some implementations relate to training the meta-learning model using imitation learning based on one or more human guided demonstrations of the task. Additional or alternative implementations relate to training the meta-learning model using reinforcement learning based on trials of the robot attempting to perform the task. Further implementations relate to using the trained meta-learning model to few shot (or one shot) learn a new task based on a human guided demonstration of the new task.