公开(公告)号：US12051206B2

公开(公告)日：2024-07-30

申请号：US16938706

申请日：2020-07-24

Applicant: NVIDIA Corporation

Inventor： Ke Chen ,  Nikolai Smolyanskiy ,  Alexey Kamenev ,  Ryan Oldja ,  Tilman Wekel ,  David Nister ,  Joachim Pehserl ,  Ibrahim Eden ,  Sangmin Oh ,  Ruchi Bhargava

IPC: G06T7/00 ,  G05D1/00 ,  G06F18/00 ,  G06F18/22 ,  G06F18/23 ,  G06T5/50 ,  G06T7/10 ,  G06T7/11 ,  G06V10/82 ,  G06V20/56 ,  G06V20/58 ,  G06V10/44

CPC classification number: G06T7/11 ,  G05D1/0088 ,  G06F18/22 ,  G06F18/23 ,  G06T5/50 ,  G06T7/10 ,  G06V10/82 ,  G06V20/56 ,  G06V20/58 ,  G06T2207/10028 ,  G06T2207/20084 ,  G06T2207/30252 ,  G06V10/454

Abstract: A deep neural network(s) (DNN) may be used to perform panoptic segmentation by performing pixel-level class and instance segmentation of a scene using a single pass of the DNN. Generally, one or more images and/or other sensor data may be stitched together, stacked, and/or combined, and fed into a DNN that includes a common trunk and several heads that predict different outputs. The DNN may include a class confidence head that predicts a confidence map representing pixels that belong to particular classes, an instance regression head that predicts object instance data for detected objects, an instance clustering head that predicts a confidence map of pixels that belong to particular instances, and/or a depth head that predicts range values. These outputs may be decoded to identify bounding shapes, class labels, instance labels, and/or range values for detected objects, and used to enable safe path planning and control of an autonomous vehicle.

公开(公告)号：US11941819B2

公开(公告)日：2024-03-26

申请号：US17457825

申请日：2021-12-06

Applicant: NVIDIA Corporation

Inventor： Dongwoo Lee ,  Junghyun Kwon ,  Sangmin Oh ,  Wenchao Zheng ,  Hae-Jong Seo ,  David Nister ,  Berta Rodriguez Hervas

IPC: G06T7/13 ,  G06T7/40 ,  G06T17/30 ,  G06V10/44 ,  G06V10/75 ,  G06V10/772 ,  G06V10/82 ,  G06V20/58

CPC classification number: G06T7/13 ,  G06T7/40 ,  G06T17/30 ,  G06V10/454 ,  G06V10/751 ,  G06V10/772 ,  G06V10/82 ,  G06V20/586 ,  G06T2207/10021 ,  G06T2207/20084 ,  G06T2207/30264

Abstract: A neural network may be used to determine corner points of a skewed polygon (e.g., as displacement values to anchor box corner points) that accurately delineate a region in an image that defines a parking space. Further, the neural network may output confidence values predicting likelihoods that corner points of an anchor box correspond to an entrance to the parking spot. The confidence values may be used to select a subset of the corner points of the anchor box and/or skewed polygon in order to define the entrance to the parking spot. A minimum aggregate distance between corner points of a skewed polygon predicted using the CNN(s) and ground truth corner points of a parking spot may be used simplify a determination as to whether an anchor box should be used as a positive sample for training.

公开(公告)号：US11885907B2

公开(公告)日：2024-01-30

申请号：US16836583

申请日：2020-03-31

Applicant: NVIDIA Corporation

Inventor： Alexander Popov ,  Nikolai Smolyanskiy ,  Ryan Oldja ,  Shane Murray ,  Tilman Wekel ,  David Nister ,  Joachim Pehserl ,  Ruchi Bhargava ,  Sangmin Oh

IPC: G01S7/295 ,  G06T7/246 ,  G06T7/73 ,  G01S7/41 ,  G01S13/931 ,  G06N3/08 ,  G06V10/764 ,  G06V10/82 ,  G06V20/58 ,  G06V20/64

CPC classification number: G01S7/2955 ,  G01S7/414 ,  G01S7/417 ,  G01S13/931 ,  G06N3/08 ,  G06T7/246 ,  G06T7/73 ,  G06V10/764 ,  G06V10/82 ,  G06V20/58 ,  G06V20/64 ,  G06T2207/10044 ,  G06T2207/20084 ,  G06T2207/30261

Abstract: In various examples, a deep neural network(s) (e.g., a convolutional neural network) may be trained to detect moving and stationary obstacles from RADAR data of a three dimensional (3D) space, in both highway and urban scenarios. RADAR detections may be accumulated, ego-motion-compensated, orthographically projected, and fed into a neural network(s). The neural network(s) may include a common trunk with a feature extractor and several heads that predict different outputs such as a class confidence head that predicts a confidence map and an instance regression head that predicts object instance data for detected objects. The outputs may be decoded, filtered, and/or clustered to form bounding shapes identifying the location, size, and/or orientation of detected object instances. The detected object instances may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

公开(公告)号：US20230360232A1

公开(公告)日：2023-11-09

申请号：US17955827

申请日：2022-09-29

Applicant: NVIDIA Corporation

Inventor： Mehmet K. Kocamaz ,  Parthiv Parikh ,  Sangmin Oh

IPC: G06T7/246

CPC classification number: G06T7/248 ,  G06T2207/30261

Abstract: In various examples, systems and methods for tracking objects and determining time-to-collision values associated with the objects are described. For instance, the systems and methods may use feature points associated with an object depicted in a first image and feature points associated with a second image to determine a scalar change associated with the object. The systems and methods may then use the scalar change to determine a translation associated with the object. Using the scalar change and the translation, the systems and methods may determine that the object is also depicted in the second image. The systems and methods may further use the scalar change and a temporal baseline to determine a time-to-collision associated with the object. After performing the determinations, the systems and methods may output data representing at least an identifier for the object, a location of the object, and/or the time-to-collision.

公开(公告)号：US20230294727A1

公开(公告)日：2023-09-21

申请号：US17695621

申请日：2022-03-15

Applicant: NVIDIA Corporation

Inventor： Sangmin Oh ,  Baris Evrim Demiroz ,  Gang Pan ,  Dong Zhang ,  Joachim Pehserl ,  Samuel Rupp Ogden ,  Tae Eun Choe

IPC: B60W60/00 ,  G06K9/62 ,  G06F9/50

CPC classification number: B60W60/001 ,  G06K9/6288 ,  G06F9/5072 ,  B60W2555/20 ,  B60W2420/42 ,  B60W2420/52

Abstract: In various examples, a hazard detection system plots hazard indicators from multiple detection sensors to grid cells of an occupancy grid corresponding to a driving environment. For example, as the ego-machine travels along a roadway, one or more sensors of the ego-machine may capture sensor data representing the driving environment. A system of the ego-machine may then analyze the sensor data to determine the existence and/or location of the one or more hazards within an occupancy grid—and thus within the environment. When a hazard is detected using a respective sensor, the system may plot an indicator of the hazard to one or more grid cells that correspond to the detected location of the hazard. Based, at least in part, on a fused or combined confidence of the hazard indicators for each grid cell, the system may predict whether the corresponding grid cell is occupied by a hazard.

公开(公告)号：US20230145218A1

公开(公告)日：2023-05-11

申请号：US17454338

申请日：2021-11-10

Applicant: NVIDIA Corporation

Inventor： Shane Murray ,  Sangmin Oh

IPC: B60W60/00 ,  G01S13/931 ,  G06N3/08

CPC classification number: B60W60/0015 ,  G01S13/931 ,  G06N3/08 ,  B60W2554/801 ,  B60W2554/802 ,  B60W2554/402 ,  B60W2554/20 ,  B60W2556/60 ,  B60W2420/52

Abstract: In various examples, systems are described herein that may evaluate one or more radar detections against a set of filter criteria, the one or more radar detections generated using at least one sensor of a vehicle. The system may then accumulate, based at least on the evaluating, the one or more radar detections to one or energy levels that correspond to one or more locations of the one or more radar detections in a zone positioned relative to the vehicle. The system may then determine one or more safety statuses associated with the zone based at least on one or more magnitudes of the one or more energy levels. The system may transmit data, or take some other action, that causes control of the vehicle based at least on the one or more safety statuses.

公开(公告)号：US20230049567A1

公开(公告)日：2023-02-16

申请号：US17976581

申请日：2022-10-28

Applicant: NVIDIA Corporation

Inventor： Alexander Popov ,  Nikolai Smolyanskiy ,  Ryan Oldja ,  Shane Murray ,  Tilman Wekel ,  David Nister ,  Joachim Pehserl ,  Ruchi Bhargava ,  Sangmin Oh

IPC: G01S7/41 ,  G01S13/86 ,  G06N3/04 ,  G06N3/08 ,  G01S13/89

Abstract: In various examples, a deep neural network(s) (e.g., a convolutional neural network) may be trained to detect moving and stationary obstacles from RADAR data of a three dimensional (3D) space. In some embodiments, ground truth training data for the neural network(s) may be generated from LIDAR data. More specifically, a scene may be observed with RADAR and LIDAR sensors to collect RADAR data and LIDAR data for a particular time slice. The RADAR data may be used for input training data, and the LIDAR data associated with the same or closest time slice as the RADAR data may be annotated with ground truth labels identifying objects to be detected. The LIDAR labels may be propagated to the RADAR data, and LIDAR labels containing less than some threshold number of RADAR detections may be omitted. The (remaining) LIDAR labels may be used to generate ground truth data.

公开(公告)号：US20220318559A1

公开(公告)日：2022-10-06

申请号：US17356224

申请日：2021-06-23

Applicant: NVIDIA Corporation

Inventor： Yichun Shen ,  Wanli Jiang ,  Junghyun Kwon ,  Siyi Li ,  Sangmin Oh ,  Minwoo Park

IPC: G06K9/00 ,  G06T7/194 ,  G06N3/04 ,  G06N20/00 ,  G06K9/66

Abstract: Apparatuses, systems, and techniques to identify bounding boxes of objects with in an image. In at least one embodiment, bounding boxes are determined in an image using an intersection over union threshold that is based at least in part on a size of an object.

公开(公告)号：US20220222480A1

公开(公告)日：2022-07-14

申请号：US17160271

申请日：2021-01-27

Applicant: NVIDIA Corporation

Inventor： Wanli Jiang ,  Yichun Shen ,  Junghyun Kwon ,  Siyi Li ,  Sangmin Oh ,  Minwoo Park

IPC: G06K9/20 ,  G06K9/00 ,  G06N3/08

Abstract: Apparatuses, systems, and techniques to generate bounding box information. In at least one embodiment, for example, bounding box information is generated based, at least in part, on a plurality of candidate bounding box information.

公开(公告)号：US20210063578A1

公开(公告)日：2021-03-04

申请号：US17005788

申请日：2020-08-28

Applicant: NVIDIA Corporation

Inventor： Tilman Wekel ,  Sangmin Oh ,  David Nister ,  Joachim Pehserl ,  Neda Cvijetic ,  Ibrahim Eden

IPC: G01S17/894 ,  G01S17/931 ,  G01S7/481

Abstract: In various examples, a deep neural network (DNN) may be used to detect and classify animate objects and/or parts of an environment. The DNN may be trained using camera-to-LiDAR cross injection to generate reliable ground truth data for LiDAR range images. For example, annotations generated in the image domain may be propagated to the LiDAR domain to increase the accuracy of the ground truth data in the LiDAR domain—e.g., without requiring manual annotation in the LiDAR domain. Once trained, the DNN may output instance segmentation masks, class segmentation masks, and/or bounding shape proposals corresponding to two-dimensional (2D) LiDAR range images, and the outputs may be fused together to project the outputs into three-dimensional (3D) LiDAR point clouds. This 2D and/or 3D information output by the DNN may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.