公开(公告)号：US20210390371A1

公开(公告)日：2021-12-16

申请号：US17460861

申请日：2021-08-30

Applicant: X DEVELOPMENT LLC

Inventor： Krishna Shankar ,  Nicolas Hudson ,  Alexander Toshev

IPC: G06N3/04 ,  G06N3/08 ,  B25J9/16 ,  G06N3/00 ,  G06K9/62

Abstract: Methods, apparatus, and computer-readable media for determining and utilizing corrections to robot actions. Some implementations are directed to updating a local features model of a robot in response to determining a human correction of an action performed by the robot. The local features model is used to determine, based on an embedding generated over a corresponding neural network model, one or more features that are most similar to the generated embedding. Updating the local features model in response to a human correction can include updating a feature embedding, of the local features model, that corresponds to the human correction. Adjustment(s) to the features model can immediately improve robot performance without necessitating retraining of the corresponding neural network model.

公开(公告)号：US10861184B1

公开(公告)日：2020-12-08

申请号：US15410702

申请日：2017-01-19

Applicant: X Development LLC

Inventor： Mrinal Kalakrishnan ,  Adrian Ling Hin Li ,  Nicolas Hudson

IPC: G06T7/73 ,  G06T7/60 ,  G06K9/66 ,  G06T7/11 ,  G06K9/52

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium for predicting object pose. In one aspect, a method includes receiving an image of an object having one or more feature points; providing the image as an input to a neural network subsystem trained to receive images of objects and to generate an output including a heat map for each feature point; applying a differentiable transformation on each heat map to generate respective one or more feature coordinates for each feature point; providing the feature coordinates for each feature point as input to an object pose solver configured to compute a predicted object pose for the object, wherein the predicted object pose for the object specifies a position and an orientation of an object; and receiving, at the output of the object pose solver, a predicted object pose for the object in the image.

公开(公告)号：US11475291B2

公开(公告)日：2022-10-18

申请号：US15855329

申请日：2017-12-27

Applicant: X Development LLC

Inventor： Nareshkumar Rajkumar ,  Patrick Leger ,  Nicolas Hudson ,  Krishna Shankar ,  Rainer Hessmer

IPC: G06F13/00 ,  G06N3/08 ,  B25J9/00 ,  B25J9/16 ,  H04L67/12 ,  H04L67/568

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for sharing learned information among robots. In some implementations, a robot obtains sensor data indicating characteristics of an object. The robot determines a classification for the object and generates an embedding for the object using a machine learning model stored by the robot. The robot stores the generated embedding and data indicating the classification for the object. The robot sends the generated embedding and the data indicating the classification to a server system. The robot receives, from the server system, an embedding generated by a second robot and a corresponding classification. The robot stores the received embedding and the corresponding classification in the local cache of the robot. The robot may then use the information in the cache to identify objects.

公开(公告)号：US11198217B2

公开(公告)日：2021-12-14

申请号：US16728159

申请日：2019-12-27

Applicant: X Development LLC

Inventor： Nicolas Hudson ,  Devesh Yamparala

IPC: B25J13/08 ,  B25J9/16 ,  G05B13/02 ,  G06N3/04 ,  G06N3/08 ,  G06N3/00

Abstract: Methods, apparatus, and computer-readable media for determining and utilizing human corrections to robot actions. In some implementations, in response to determining a human correction of a robot action, a correction instance is generated that includes sensor data, captured by one or more sensors of the robot, that is relevant to the corrected action. The correction instance can further include determined incorrect parameter(s) utilized in performing the robot action and/or correction information that is based on the human correction. The correction instance can be utilized to generate training example(s) for training one or model(s), such as neural network model(s), corresponding to those used in determining the incorrect parameter(s). In various implementations, the training is based on correction instances from multiple robots. After a revised version of a model is generated, the revised version can thereafter be utilized by one or more of the multiple robots.

公开(公告)号：US20240078683A1

公开(公告)日：2024-03-07

申请号：US18295998

申请日：2023-04-05

Applicant: X Development LLC

Inventor： Mrinal Kalakrishnan ,  Adrian Ling Hin Li ,  Nicolas Hudson

IPC: G06T7/246 ,  G06T7/11 ,  G06T7/60 ,  G06T7/73 ,  G06V10/42 ,  G06V10/75 ,  G06V10/82 ,  G06V20/10 ,  G06V30/186 ,  G06V30/19 ,  G06V30/194

CPC classification number: G06T7/246 ,  G06T7/11 ,  G06T7/60 ,  G06T7/73 ,  G06V10/42 ,  G06V10/758 ,  G06V10/82 ,  G06V20/10 ,  G06V30/186 ,  G06V30/19173 ,  G06V30/194 ,  G06T2207/10004 ,  G06T2207/20084 ,  G06T2207/30244

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium for predicting object pose. In one aspect, a method includes receiving an image of an object having one or more feature points; providing the image as an input to a neural network subsystem trained to receive images of objects and to generate an output including a heat map for each feature point; applying a differentiable transformation on each heat map to generate respective one or more feature coordinates for each feature point; providing the feature coordinates for each feature point as input to an object pose solver configured to compute a predicted object pose for the object, wherein the predicted object pose for the object specifies a position and an orientation of an object; and receiving, at the output of the object pose solver, a predicted object pose for the object in the image.

公开(公告)号：US11640517B2

公开(公告)日：2023-05-02

申请号：US17460861

申请日：2021-08-30

Applicant: X DEVELOPMENT LLC

Inventor： Krishna Shankar ,  Nicolas Hudson ,  Alexander Toshev

IPC: G06N3/04 ,  G06N3/08 ,  B25J9/16 ,  G06K9/62 ,  G06N3/00 ,  G06N3/008 ,  G06N3/084

Abstract: Methods, apparatus, and computer-readable media for determining and utilizing corrections to robot actions. Some implementations are directed to updating a local features model of a robot in response to determining a human correction of an action performed by the robot. The local features model is used to determine, based on an embedding generated over a corresponding neural network model, one or more features that are most similar to the generated embedding. Updating the local features model in response to a human correction can include updating a feature embedding, of the local features model, that corresponds to the human correction. Adjustment(s) to the features model can immediately improve robot performance without necessitating retraining of the corresponding neural network model.

公开(公告)号：US20230004802A1

公开(公告)日：2023-01-05

申请号：US17930874

申请日：2022-09-09

Applicant: X Development LLC

Inventor： Nareshkumar Rajkumar ,  Patrick Leger ,  Nicolas Hudson ,  Krishna Shankar ,  Rainer Hessmer

IPC: G06N3/08 ,  B25J9/00 ,  B25J9/16

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for sharing learned information among robots. In some implementations, a robot obtains sensor data indicating characteristics of an object. The robot determines a classification for the object and generates an embedding for the object using a machine learning model stored by the robot. The robot stores the generated embedding and data indicating the classification for the object. The robot sends the generated embedding and the data indicating the classification to a server system. The robot receives, from the server system, an embedding generated by a second robot and a corresponding classification. The robot stores the received embedding and the corresponding classification in the local cache of the robot. The robot may then use the information in the cache to identify objects.

公开(公告)号：US20220055209A1

公开(公告)日：2022-02-24

申请号：US17520175

申请日：2021-11-05

Applicant: X Development LLC

Inventor： Nicolas Hudson ,  Devesh Yamparala

IPC: B25J9/16 ,  G05B13/02 ,  G06N3/04 ,  G06N3/08 ,  G06N3/00

Abstract: Methods, apparatus, and computer-readable media for determining and utilizing human corrections to robot actions. In some implementations, in response to determining a human correction of a robot action, a correction instance is generated that includes sensor data, captured by one or more sensors of the robot, that is relevant to the corrected action. The correction instance can further include determined incorrect parameter(s) utilized in performing the robot action and/or correction information that is based on the human correction. The correction instance can be utilized to generate training example(s) for training one or model(s), such as neural network model(s), corresponding to those used in determining the incorrect parameter(s). In various implementations, the training is based on correction instances from multiple robots. After a revised version of a model is generated, the revised version can thereafter be utilized by one or more of the multiple robots.

公开(公告)号：US20190197396A1

公开(公告)日：2019-06-27

申请号：US15855329

申请日：2017-12-27

Applicant: X Development LLC

Inventor： Nareshkumar Rajkumar ,  Patrick Leger ,  Nicolas Hudson ,  Krishna Shankar ,  Rainer Hessmer

IPC: G06N3/08

CPC classification number: G06N3/08 ,  B25J9/0003 ,  B25J9/1671 ,  G05B2219/45108 ,  H04L67/12 ,  H04L67/2842

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for sharing learned information among robots. In some implementations, a robot obtains sensor data indicating characteristics of an object. The robot determines a classification for the object and generates an embedding for the object using a machine learning model stored by the robot. The robot stores the generated embedding and data indicating the classification for the object. The robot sends the generated embedding and the data indicating the classification to a server system. The robot receives, from the server system, an embedding generated by a second robot and a corresponding classification. The robot stores the received embedding and the corresponding classification in the local cache of the robot. The robot may then use the information in the cache to identify objects.

公开(公告)号：US11625852B1

公开(公告)日：2023-04-11

申请号：US17114083

申请日：2020-12-07

Applicant: X Development LLC

Inventor： Mrinal Kalakrishnan ,  Adrian Ling Hin Li ,  Nicolas Hudson

IPC: G06T7/73 ,  G06T7/60 ,  G06T7/11 ,  G06V10/42 ,  G06V30/194

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium for predicting object pose. In one aspect, a method includes receiving an image of an object having one or more feature points; providing the image as an input to a neural network subsystem trained to receive images of objects and to generate an output including a heat map for each feature point; applying a differentiable transformation on each heat map to generate respective one or more feature coordinates for each feature point; providing the feature coordinates for each feature point as input to an object pose solver configured to compute a predicted object pose for the object, wherein the predicted object pose for the object specifies a position and an orientation of an object; and receiving, at the output of the object pose solver, a predicted object pose for the object in the image.