公开(公告)号：US20230294687A1

公开(公告)日：2023-09-21

申请号：US18169105

申请日：2023-02-14

Applicant: Waymo LLC

Inventor： James Philbin ,  Vasiliy Igorevich Karasev ,  Alper Ayvaci ,  Marc Wimmershoff ,  Dragomir Dimitrov Anguelov

IPC: B60W30/095 ,  G05B13/02

CPC classification number: B60W30/0956 ,  G05B13/027 ,  B60W2420/42 ,  B60W2420/52 ,  B60W2554/4044 ,  B60W2555/60 ,  B60W2556/40

Abstract: The described aspects and implementations enable efficient object detection and tracking. In one implementation, disclosed is a method and a system to perform the method, the system including the sensing system configured to obtain sensing data characterizing an environment of the vehicle. The system further includes a data processing system operatively coupled to the sensing system and configured to process the sensing data using a first (second) set of neural network (NN) layers to obtain a first (second) set of features for a first (second) region of the environment, the first (second) set of features is associated with a first (second) spatial resolution. The data processing system is further to process the two sets of features using a second set of NN layers to detect a location of obj ect(s) in the environment of the vehicle and a state of motion of the object(s).

公开(公告)号：US20210294346A1

公开(公告)日：2021-09-23

申请号：US17343187

申请日：2021-06-09

Applicant: Waymo LLC

Inventor： Junhua Mao ,  Congcong Li ,  Alper Ayvaci ,  Chen Sun ,  Kevin Murphy ,  Ruichi Yu

IPC: G05D1/02 ,  G05D1/00 ,  G06K9/00 ,  G06K9/62 ,  B60W30/095 ,  G01S17/931

Abstract: Aspects of the disclosure relate to training and using a model for identifying actions of objects. For instance, LIDAR sensor data frames including an object bounding box corresponding to an object as well as an action label for the bounding box may be received. Each sensor frame is associated with a timestamp and is sequenced with respect to other sensor frames. Each given sensor data frame may be projected into a camera image of the object based on the timestamp associated with the given sensor data frame in order to provide fused data. The model may be trained using the fused data such that the model is configured to, in response to receiving fused data, the model outputs an action label for each object bounding box of the fused data. This output may then be used to control a vehicle in an autonomous driving mode.

公开(公告)号：US11669980B2

公开(公告)日：2023-06-06

申请号：US17384654

申请日：2021-07-23

Applicant: Waymo LLC

Inventor： Daniel Rudolf Maurer ,  Alper Ayvaci ,  Nichola Abdo ,  Christopher John Sweeney ,  Robert William Anderson

IPC: G06T7/269 ,  G06T7/246

CPC classification number: G06T7/269 ,  G06T7/248 ,  G06T2207/10028 ,  G06T2207/20084 ,  G06T2207/30261

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating motion detection based on optical flow. One of the methods includes obtaining a first image of a scene in an environment taken by an agent at a first time point and a second image of the scene at a second later time point. A point cloud characterizing the scene in the environment is obtained. A predicted optical flow is determined between the first image and the second image. A respective initial flow prediction for the point that represents motion of the point between the two time points is determined. A respective ego motion flow estimate for the point that represents a motion of the point induced by ego motion of the agent is determined. A respective motion prediction that indicates whether the point was static or in motion between the two time points is determined.

公开(公告)号：US20230046289A1

公开(公告)日：2023-02-16

申请号：US17893376

申请日：2022-08-23

Applicant: WAYMO LLC

Inventor： Justin Thorsen ,  Changchang Wu ,  Alper Ayvaci ,  Tiffany Chen ,  Lo Po Tsui ,  Zhinan Xu ,  Chen Wu ,  Sean Rafferty

IPC: G06K19/067 ,  G09F3/00

Abstract: Aspects of the disclosure provide for automatically generating labels for sensor data. For instance, first sensor data, for a vehicle may be identified. This first sensor data may have been captured by a first sensor of the vehicle at a first location during a first point in time and may be associated with a first label for an object. Second sensor data for the vehicle may be identified. The second sensor data may have been captured by a second sensor of the vehicle at a second location at a second point in time outside of the first point in time. The second location is different from the first location. A determination may be made as to whether the object is a static object. Based on the determination that the object is a static object, the first label may be used to automatically generate a second label for the second sensor data.

公开(公告)号：US20220343657A1

公开(公告)日：2022-10-27

申请号：US17862499

申请日：2022-07-12

Applicant: Waymo LLC

Inventor： Hyman Murveit ,  Abhijit Ogale ,  Yu-Han Chen ,  Alper Ayvaci

IPC: G06V20/58 ,  G06T7/174 ,  G06K9/62 ,  G05D1/00 ,  G05D1/02 ,  G01S13/931 ,  G01S7/41

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a future occupancy prediction for a region of an environment. In one aspect, a method comprises: receiving sensor data generated by a sensor system of a vehicle that characterizes an environment in a vicinity of the vehicle as of a current time point, wherein the sensor data comprises a plurality of sensor samples characterizing the environment that were each captured at different time points; processing a network input comprising the sensor data using a neural network to generate an occupancy prediction output for a region of the environment, wherein: the occupancy prediction output characterizes, for one or more future intervals of time after the current time point, a respective likelihood that the region of the environment will be occupied by an agent in the environment during the future interval of time.

公开(公告)号：US11475263B2

公开(公告)日：2022-10-18

申请号：US16827835

申请日：2020-03-24

Applicant: WAYMO LLC

Inventor： Justin Thorsen ,  Changchang Wu ,  Alper Ayvaci ,  Tiffany Chen ,  Lo Po Tsui ,  Zhinan Xu ,  Chen Wu ,  Sean Rafferty

IPC: G06K19/067 ,  G09F3/00

Abstract: Aspects of the disclosure provide for automatically generating labels for sensor data. For instance, first sensor data for a vehicle may be identified. This first sensor data may have been captured by a first sensor of the vehicle at a first location during a first point in time and may be associated with a first label for an object. Second sensor data for the vehicle may be identified. The second sensor data may have been captured by a second sensor of the vehicle at a second location at a second point in time outside of the first point in time. The second location is different from the first location. A determination may be made as to whether the object is a static object. Based on the determination that the object is a static object, the first label may be used to automatically generate a second label for the second sensor data.

公开(公告)号：US12266190B2

公开(公告)日：2025-04-01

申请号：US17884356

申请日：2022-08-09

Applicant: Waymo LLC

Inventor： Albert Zhao ,  Vasiliy Igorevich Karasev ,  Hang Yan ,  Daniel Rudolf Maurer ,  Alper Ayvaci ,  Yu-Han Chen

IPC: G06V20/58 ,  G06T7/55 ,  G06V10/44 ,  G06V10/82 ,  G06T3/4046 ,  G06V10/40 ,  G06V10/70 ,  G06V20/69 ,  G06V30/18

Abstract: The described aspects and implementations enable efficient detection and classification of objects with machine learning models that deploy a bird's-eye view representation and are trained using depth ground truth data. In one implementation, disclosed are system and techniques that include obtaining images, generating, using a first neural network (NN), feature vectors (FVs) and depth distributions pixels of images, wherein the first NN is trained using training images and a depth ground truth data for the training images. The techniques further include obtaining a feature tensor (FT) in view of the FVs and the depth distributions, and processing the obtained FTs, using a second NN, to identify one or more objects depicted in the images.

公开(公告)号：US20240096105A1

公开(公告)日：2024-03-21

申请号：US17884356

申请日：2022-08-09

Applicant: Waymo LLC

Inventor： Albert Zhao ,  Vasiliy Igorevich Karasev ,  Hang Yan ,  Daniel Rudolf Maurer ,  Alper Ayvaci ,  Yu-Han Chen

IPC: G06V20/58 ,  G06T7/55 ,  G06V10/44 ,  G06V10/82

CPC classification number: G06V20/58 ,  G06T7/55 ,  G06V10/44 ,  G06V10/82 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/30252

Abstract: The described aspects and implementations enable efficient detection and classification of objects with machine learning models that deploy a bird's-eye view representation and are trained using depth ground truth data. In one implementation, disclosed are system and techniques that include obtaining images, generating, using a first neural network (NN), feature vectors (FVs) and depth distributions pixels of images, wherein the first NN is trained using training images and a depth ground truth data for the training images. The techniques further include obtaining a feature tensor (FT) in view of the FVs and the depth distributions, and processing the obtained FTs, using a second NN, to identify one or more objects depicted in the images.

公开(公告)号：US20230033989A1

公开(公告)日：2023-02-02

申请号：US17384654

申请日：2021-07-23

Applicant: Waymo LLC

Inventor： Daniel Rudolf Maurer ,  Alper Ayvaci ,  Nichola Abdo ,  Christopher John Sweeney ,  Robert William Anderson

IPC: G06T7/269 ,  G06T7/246

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating motion detection based on optical flow. One of the methods includes obtaining a first image of a scene in an environment taken by an agent at a first time point and a second image of the scene at a second later time point. A point cloud characterizing the scene in the environment is obtained. A predicted optical flow is determined between the first image and the second image. A respective initial flow prediction for the point that represents motion of the point between the two time points is determined. A respective ego motion flow estimate for the point that represents a motion of the point induced by ego motion of the agent is determined. A respective motion prediction that indicates whether the point was static or in motion between the two time points is determined.

公开(公告)号：US11544869B2

公开(公告)日：2023-01-03

申请号：US17342434

申请日：2021-06-08

Applicant: Waymo LLC

Inventor： Alper Ayvaci ,  Yu-Han Chen ,  Ruichi Yu ,  Chen Wu ,  Noha Waheed Ahmed Radwan ,  Jonathon Shlens

IPC: G06T7/73 ,  G06T7/50 ,  G06K9/62 ,  G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating object interaction predictions using a neural network. One of the methods includes obtaining a sensor input derived from data generated by one or more sensors that characterizes a scene. The sensor input is provided to an object interaction neural network. The object interaction neural network is configured to process the sensor input to generate a plurality of object interaction outputs. Each respective object interaction output includes main object information and interacting object information. The respective object interaction outputs corresponding to the plurality of regions in the sensor input are received as output of the object interaction neural network.