公开(公告)号：US12134199B2

公开(公告)日：2024-11-05

申请号：US17642325

申请日：2020-09-09

Applicant: GOOGLE LLC

Inventor： Soeren Pirk ,  Seyed Mohammad Khansari Zadeh ,  Karol Hausman ,  Alexander Toshev

IPC: B25J9/16 ,  G06N3/045 ,  G06V10/147 ,  G06V10/44 ,  G06V10/82 ,  G06V20/10 ,  G06V20/17 ,  G06V20/13

Abstract: Training and/or using a machine learning model for performing robotic tasks is disclosed herein. In many implementations, an environment-conditioned action sequence prediction model is used to determine a set of actions as well as a corresponding particular order for the actions for the robot to perform to complete the task. In many implementations, each action in the set of actions has a corresponding action network used to control the robot in performing the action.

公开(公告)号：US12112494B2

公开(公告)日：2024-10-08

申请号：US17053335

申请日：2020-02-28

Applicant: Google LLC

Inventor： Honglak Lee ,  Xinchen Yan ,  Soeren Pirk ,  Yunfei Bai ,  Seyed Mohammad Khansari Zadeh ,  Yuanzheng Gong ,  Jasmine Hsu

IPC: G06T7/55 ,  B25J9/16 ,  B25J13/08 ,  G06F18/21 ,  G06T7/50 ,  G06V20/10 ,  G06V20/64

CPC classification number: G06T7/55 ,  B25J9/1605 ,  B25J9/163 ,  B25J9/1669 ,  B25J9/1697 ,  B25J13/08 ,  G06F18/2163 ,  G06T7/50 ,  G06V20/10 ,  G06V20/64 ,  G06T2207/10024 ,  G06T2207/10028 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/20132

Abstract: Implementations relate to training a point cloud prediction model that can be utilized to process a single-view two-and-a-half-dimensional (2.5D) observation of an object, to generate a domain-invariant three-dimensional (3D) representation of the object. Implementations additionally or alternatively relate to utilizing the domain-invariant 3D representation to train a robotic manipulation policy model using, as at least part of the input to the robotic manipulation policy model during training, the domain-invariant 3D representations of simulated objects to be manipulated. Implementations additionally or alternatively relate to utilizing the trained robotic manipulation policy model in control of a robot based on output generated by processing generated domain-invariant 3D representations utilizing the robotic manipulation policy model.

公开(公告)号：US11887363B2

公开(公告)日：2024-01-30

申请号：US17279924

申请日：2019-09-27

Applicant: Google LLC

Inventor： Soeren Pirk ,  Yunfei Bai ,  Pierre Sermanet ,  Seyed Mohammad Khansari Zadeh ,  Harrison Lynch

IPC: G06V20/10 ,  B25J9/16 ,  B25J13/00 ,  G05B13/02 ,  G06N3/08 ,  G10L15/22 ,  G06F18/21 ,  G06F18/2413 ,  G06V10/764 ,  G06V10/70 ,  G06V10/776 ,  G06V10/82

CPC classification number: G06V20/10 ,  B25J9/1697 ,  B25J13/003 ,  G05B13/027 ,  G06F18/217 ,  G06F18/2413 ,  G06N3/08 ,  G06V10/764 ,  G06V10/768 ,  G06V10/776 ,  G06V10/82 ,  G10L15/22

Abstract: Training a machine learning model (e.g., a neural network model such as a convolutional neural network (CNN) model) so that, when trained, the model can be utilized in processing vision data (e.g., from a vision component of a robot), that captures an object, to generate a rich object-centric embedding for the vision data. The generated embedding can enable differentiation of even subtle variations of attributes of the object captured by the vision data.

公开(公告)号：US20230419521A1

公开(公告)日：2023-12-28

申请号：US18367888

申请日：2023-09-13

Applicant: Google LLC

Inventor： Vincent Michael Casser ,  Soeren Pirk ,  Reza Mahjourian ,  Anelia Angelova

IPC: G06T7/55 ,  G06T7/246 ,  G06N3/088 ,  G06T3/00 ,  G06N3/045

CPC classification number: G06T7/55 ,  G06T7/248 ,  G06N3/088 ,  G06T3/0093 ,  G06N3/045 ,  G06T2207/20081 ,  G06T2207/20084

Abstract: A system for generating a depth output for an image is described. The system receives input images that depict the same scene, each input image including one or more potential objects. The system generates, for each input image, a respective background image and processes the background images to generate a camera motion output that characterizes the motion of the camera between the input images. For each potential object, the system generates a respective object motion output for the potential object based on the input images and the camera motion output. The system processes a particular input image of the input images using a depth prediction neural network (NN) to generate a depth output for the particular input image, and updates the current values of parameters of the depth prediction NN based on the particular depth output, the camera motion output, and the object motion outputs for the potential objects.

公开(公告)号：US20210279511A1

公开(公告)日：2021-09-09

申请号：US17194090

申请日：2021-03-05

Applicant: Google LLC

Inventor： Ariel Gordon ,  Soeren Pirk ,  Anelia Angelova ,  Vincent Michael Casser ,  Yao Lu ,  Anthony Brohan ,  Zhao Chen ,  Jan Dlabal

IPC: G06K9/62 ,  G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network using consistency measures. One of the methods includes processing a particular training example from a mediator training data set using a first neural network to generate a first output for a first machine learning task; processing the particular training example in the mediator training data set using each of one or more second neural networks, wherein each second neural network is configured to generate a second output for a respective second machine learning task; determining, for each second machine learning task, a consistency target output for the first machine learning task; determining, for each second machine learning task, an error between the first output and the consistency target output corresponding to the second machine learning task; and generating a parameter update for the first neural network from the determined errors.

公开(公告)号：US20210073997A1

公开(公告)日：2021-03-11

申请号：US16562819

申请日：2019-09-06

Applicant: Google LLC

Inventor： Suhani Vora ,  Reza Mahjourian ,  Soeren Pirk ,  Anelia Angelova

IPC: G06T7/11 ,  G06T7/246 ,  G06T7/55 ,  G06K9/00 ,  G06N3/08

Abstract: This disclosure describes a system including one or more computers and one or more non-transitory storage devices storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for generating a predicted segmentation map for potential objects in a future scene depicted in a future image. The operations includes: receiving a sequence of input images that depict the same scene, the input images being captured by a camera at different time steps, the sequence of input images comprising a current input image and one or more input images preceding the current image in the sequence; processing the current input image to generate a segmentation map for potential objects in the current input image and a respective depth map for the current input image; generating a point cloud for the current input image using the segmentation map and the depth map of the current input image, wherein the point cloud is a 3-dimensional (3D) structure representation of the scene as depicted in the current input image; processing the sequence of input images using an ego-motion estimation neural network to generate, for each pair of two consecutive input images in the sequence, a respective ego-motion output that characterizes motion of the camera between the two consecutive input images; processing the ego-motion outputs using a future ego-motion prediction neural network to generate a future ego-motion output that is a prediction of future motion of the camera from the current input image in the sequence to a future image, wherein the future image is an image that would be captured by the camera at a future time step; processing the point cloud of the current input image and the future ego-motion output to generate a future point cloud that is a predicted 3D representation of a future scene as depicted in the future image; and processing the future point cloud to generate a predicted segmentation map for potential objects in the future scene depicted in the future image.