公开(公告)号：US11926346B2

公开(公告)日：2024-03-12

申请号：US17395318

申请日：2021-08-05

Applicant: NVIDIA Corporation

Inventor： Fangkai Yang ,  David Nister ,  Yizhou Wang ,  Rotem Aviv ,  Julia Ng ,  Birgit Henke ,  Hon Leung Lee ,  Yunfei Shi

IPC: B60W60/00 ,  B60W30/18 ,  G08G1/0967

CPC classification number: B60W60/0027 ,  B60W30/18154 ,  B60W30/18159 ,  G08G1/096725 ,  B60W2420/42 ,  B60W2420/52 ,  B60W2552/05

Abstract: In various examples, a yield scenario may be identified for a first vehicle. A wait element is received that encodes a first path for the first vehicle to traverse a yield area and a second path for a second vehicle to traverse the yield area. The first path is employed to determine a first trajectory in the yield area for the first vehicle based at least on a first location of the first vehicle at a time and the second path is employed to determine a second trajectory in the yield area for the second vehicle based at least on a second location of the second vehicle at the time. To operate the first vehicle in accordance with a wait state, it may be determined whether there is a conflict between the first trajectory and the second trajectory, where the wait state defines a yielding behavior for the first vehicle.

公开(公告)号：US11906660B2

公开(公告)日：2024-02-20

申请号：US17005788

申请日：2020-08-28

Applicant: NVIDIA Corporation

Inventor： Tilman Wekel ,  Sangmin Oh ,  David Nister ,  Joachim Pehserl ,  Neda Cvijetic ,  Ibrahim Eden

IPC: G01S7/00 ,  G01S7/48 ,  G01S17/894 ,  G01S7/481 ,  G01S17/931 ,  G06V10/764 ,  G06V10/80 ,  G06V10/82 ,  G06V20/58 ,  G01S7/28

CPC classification number: G01S7/4802 ,  G01S7/481 ,  G01S17/894 ,  G01S17/931 ,  G06V10/764 ,  G06V10/80 ,  G06V10/82 ,  G06V20/58 ,  G01S7/28

Abstract: In various examples, a deep neural network (DNN) may be used to detect and classify animate objects and/or parts of an environment. The DNN may be trained using camera-to-LiDAR cross injection to generate reliable ground truth data for LiDAR range images. For example, annotations generated in the image domain may be propagated to the LiDAR domain to increase the accuracy of the ground truth data in the LiDAR domain—e.g., without requiring manual annotation in the LiDAR domain. Once trained, the DNN may output instance segmentation masks, class segmentation masks, and/or bounding shape proposals corresponding to two-dimensional (2D) LiDAR range images, and the outputs may be fused together to project the outputs into three-dimensional (3D) LiDAR point clouds. This 2D and/or 3D information output by the DNN may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

公开(公告)号：US11842440B2

公开(公告)日：2023-12-12

申请号：US17228460

申请日：2021-04-12

Applicant: NVIDIA Corporation

Inventor： Philippe Bouttefroy ,  David Nister ,  Ibrahim Eden

IPC: G06T17/05 ,  G06V20/56 ,  G06V10/764 ,  G06V10/82

CPC classification number: G06T17/05 ,  G06V10/764 ,  G06V10/82 ,  G06V20/56

Abstract: In various examples, locations of directional landmarks, such as vertical landmarks, may be identified using 3D reconstruction. A set of observations of directional landmarks (e.g., images captured from a moving vehicle) may be reduced to 1D lookups by rectifying the observations to align directional landmarks along a particular direction of the observations. Object detection may be applied, and corresponding 1D lookups may be generated to represent the presence of a detected vertical landmark in an image.

公开(公告)号：US20230341234A1

公开(公告)日：2023-10-26

申请号：US17725175

申请日：2022-04-20

Applicant: NVIDIA Corporation

Inventor： David Nister ,  Hon Leung Lee ,  Yizhou Wang ,  Rotem Aviv ,  Birgit Henke ,  Julia Ng ,  Amir Akbarzadeh

IPC: G01C21/36 ,  G01C21/34 ,  B60W60/00

CPC classification number: G01C21/3658 ,  G01C21/3446 ,  G01C21/3453 ,  B60W60/001 ,  B60W2556/40

Abstract: In various examples, a lane planner for generating lane planner output data based on a state and probabilistic action space is provided. A driving system—that operates based on a hierarchical drive planning framework—includes the lane planner and other planning and control components. The lane planner processes lane planner input data (e.g., large lane graph, source node, target node) to generate lane planner output data (e.g., expected time rewards). The driving system can also include a route planner (e.g., a first planning layer) that operates to provide the lane planner input data to the lane planner. The lane planner operates as second planning layer that processes the lane planner input data based at least in part on a state and probabilistic action space of the large lane graph and calculates a time cost associated with navigating from a source node to a target node in the large lane graph.

公开(公告)号：US11790230B2

公开(公告)日：2023-10-17

申请号：US17723195

申请日：2022-04-18

Applicant: NVIDIA Corporation

Inventor： Yilin Yang ,  Bala Siva Sashank Jujjavarapu ,  Pekka Janis ,  Zhaoting Ye ,  Sangmin Oh ,  Minwoo Park ,  Daniel Herrera Castro ,  Tommi Koivisto ,  David Nister

IPC: G06K9/00 ,  G06N3/08 ,  B60W30/14 ,  B60W60/00 ,  G06V20/56 ,  G06F18/214 ,  G06V10/762

CPC classification number: G06N3/08 ,  B60W30/14 ,  B60W60/0011 ,  G06F18/2155 ,  G06V10/763 ,  G06V20/56

Abstract: In various examples, a deep neural network (DNN) is trained to accurately predict, in deployment, distances to objects and obstacles using image data alone. The DNN may be trained with ground truth data that is generated and encoded using sensor data from any number of depth predicting sensors, such as, without limitation, RADAR sensors, LIDAR sensors, and/or SONAR sensors. Camera adaptation algorithms may be used in various embodiments to adapt the DNN for use with image data generated by cameras with varying parameters—such as varying fields of view. In some examples, a post-processing safety bounds operation may be executed on the predictions of the DNN to ensure that the predictions fall within a safety-permissible range.

公开(公告)号：US11698272B2

公开(公告)日：2023-07-11

申请号：US17007873

申请日：2020-08-31

Applicant: NVIDIA Corporation

Inventor： Michael Kroepfl ,  Amir Akbarzadeh ,  Ruchi Bhargava ,  Vaibhav Thukral ,  Neda Cvijetic ,  Vadim Cugunovs ,  David Nister ,  Birgit Henke ,  Ibrahim Eden ,  Youding Zhu ,  Michael Grabner ,  Ivana Stojanovic ,  Yu Sheng ,  Jeffrey Liu ,  Enliang Zheng ,  Jordan Marr ,  Andrew Carley

IPC: G01C21/00 ,  G06N3/02 ,  G01C21/16

CPC classification number: G01C21/3841 ,  G01C21/1652 ,  G01C21/3811 ,  G01C21/3867 ,  G01C21/3878 ,  G01C21/3896 ,  G06N3/02

Abstract: An end-to-end system for data generation, map creation using the generated data, and localization to the created map is disclosed. Mapstreams—or streams of sensor data, perception outputs from deep neural networks (DNNs), and/or relative trajectory data—corresponding to any number of drives by any number of vehicles may be generated and uploaded to the cloud. The mapstreams may be used to generate map data—and ultimately a fused high definition (HD) map—that represents data generated over a plurality of drives. When localizing to the fused HD map, individual localization results may be generated based on comparisons of real-time data from a sensor modality to map data corresponding to the same sensor modality. This process may be repeated for any number of sensor modalities and the results may be fused together to determine a final fused localization result.

公开(公告)号：US20230122119A1

公开(公告)日：2023-04-20

申请号：US18067176

申请日：2022-12-16

Applicant: NVIDIA Corporation

Inventor： Yue Wu ,  Pekka Janis ,  Xin Tong ,  Cheng-Chieh Yang ,  Minwoo Park ,  David Nister

IPC: G05D1/02 ,  G06N3/04 ,  G06T7/20 ,  G05D1/00 ,  G06V20/10 ,  G06V20/58 ,  G06F18/214

Abstract: In various examples, a sequential deep neural network (DNN) may be trained using ground truth data generated by correlating (e.g., by cross-sensor fusion) sensor data with image data representative of a sequences of images. In deployment, the sequential DNN may leverage the sensor correlation to compute various predictions using image data alone. The predictions may include velocities, in world space, of objects in fields of view of an ego-vehicle, current and future locations of the objects in image space, and/or a time-to-collision (TTC) between the objects and the ego-vehicle. These predictions may be used as part of a perception system for understanding and reacting to a current physical environment of the ego-vehicle.

公开(公告)号：US20230099494A1

公开(公告)日：2023-03-30

申请号：US17489346

申请日：2021-09-29

Applicant: NVIDIA Corporation

Inventor： Mehmet Kocamaz ,  Neeraj Sajjan ,  Sangmin Oh ,  David Nister ,  Junghyun Kwon ,  Minwoo Park

IPC: G06K9/00 ,  G06K9/32 ,  G06N3/08

Abstract: In various examples, live perception from sensors of an ego-machine may be leveraged to detect objects and assign the objects to bounded regions (e.g., lanes or a roadway) in an environment of the ego-machine in real-time or near real-time. For example, a deep neural network (DNN) may be trained to compute outputs—such as output segmentation masks—that may correspond to a combination of object classification and lane identifiers. The output masks may be post-processed to determine object to lane assignments that assign detected objects to lanes in order to aid an autonomous or semi-autonomous machine in a surrounding environment.

公开(公告)号：US11604944B2

公开(公告)日：2023-03-14

申请号：US16514230

申请日：2019-07-17

Applicant: NVIDIA Corporation

Inventor： Minwoo Park ,  Xiaolin Lin ,  Hae-Jong Seo ,  David Nister ,  Neda Cvijetic

IPC: G06K9/62 ,  G06V10/75 ,  G06V20/56 ,  G06V10/44 ,  G06V10/764 ,  G06V10/776 ,  G06V10/82 ,  G06V10/94 ,  G05D1/00 ,  G06N3/04 ,  G06N3/08

Abstract: In various examples, systems and methods are disclosed that preserve rich spatial information from an input resolution of a machine learning model to regress on lines in an input image. The machine learning model may be trained to predict, in deployment, distances for each pixel of the input image at an input resolution to a line pixel determined to correspond to a line in the input image. The machine learning model may further be trained to predict angles and label classes of the line. An embedding algorithm may be used to train the machine learning model to predict clusters of line pixels that each correspond to a respective line in the input image. In deployment, the predictions of the machine learning model may be used as an aid for understanding the surrounding environment—e.g., for updating a world model—in a variety of autonomous machine applications.

公开(公告)号：US11531088B2

公开(公告)日：2022-12-20

申请号：US16836618

申请日：2020-03-31

Applicant: NVIDIA Corporation

Inventor： Alexander Popov ,  Nikolai Smolyanskiy ,  Ryan Oldja ,  Shane Murray ,  Tilman Wekel ,  David Nister ,  Joachim Pehserl ,  Ruchi Bhargava ,  Sangmin Oh

IPC: G01S7/41 ,  G01S13/86 ,  G06N3/04 ,  G06N3/08 ,  G01S13/89

Abstract: In various examples, a deep neural network(s) (e.g., a convolutional neural network) may be trained to detect moving and stationary obstacles from RADAR data of a three dimensional (3D) space. In some embodiments, ground truth training data for the neural network(s) may be generated from LIDAR data. More specifically, a scene may be observed with RADAR and LIDAR sensors to collect RADAR data and LIDAR data for a particular time slice. The RADAR data may be used for input training data, and the LIDAR data associated with the same or closest time slice as the RADAR data may be annotated with ground truth labels identifying objects to be detected. The LIDAR labels may be propagated to the RADAR data, and LIDAR labels containing less than some threshold number of RADAR detections may be omitted. The (remaining) LIDAR labels may be used to generate ground truth data.