公开(公告)号：US12175739B2

公开(公告)日：2024-12-24

申请号：US17149437

申请日：2021-01-14

Applicant: NVIDIA Corporation

Inventor： Yichun Shen ,  Wanli Jiang ,  Junghyun Kwon ,  Siyi Li ,  Minwoo Park ,  Sangmin Oh

IPC: G06V10/94 ,  G06F18/21 ,  G06V10/22 ,  G06V10/40 ,  G06V10/46

Abstract: Apparatuses, systems, and techniques to perform non-maximum suppression (NMS) in parallel to remove redundant bounding boxes. In at least one embodiment, two or more parallel circuits to perform two or more portions of a NMS algorithm in parallel to remove one or more redundant bounding boxes corresponding to one or more objects within one or more digital images.

公开(公告)号：US20240265555A1

公开(公告)日：2024-08-08

申请号：US18614160

申请日：2024-03-22

Applicant: NVIDIA Corporation

Inventor： Dong Zhang ,  Sangmin Oh ,  Junghyun Kwon ,  Baris Evrim Demiroz ,  Tae Eun Choe ,  Minwoo Park ,  Chethan Ningaraju ,  Hao Tsui ,  Eric Viscito ,  Jagadeesh Sankaran ,  Yongqing Liang

IPC: G06T7/246 ,  B60W60/00 ,  G06F18/214 ,  G06N3/08 ,  G06V10/25 ,  G06V10/75 ,  G06V20/56 ,  G06V20/58

CPC classification number: G06T7/246 ,  B60W60/001 ,  G06F18/2148 ,  G06N3/08 ,  G06V10/25 ,  G06V10/751 ,  G06V20/58 ,  G06V20/56

Abstract: Systems and methods are disclosed that use a geometric approach to detect objects on a road surface. A set of points within a region of interest between a first frame and a second frame are captured and tracked to determine a difference in location between the set of points in two frames. The first frame may be aligned with the second frame and the first pixel values of the first frame may be compared with the second pixel values of the second frame to generate a disparity image including third pixels. Subsets of the third pixels that have an disparity image value about a first threshold may be combined, and the third pixels may be scored and associated with disparity values for each pixel of the one or more subsets of the third pixels. A bounding shape may be generated based on the scoring that corresponds to the object.

公开(公告)号：US20240135173A1

公开(公告)日：2024-04-25

申请号：US18343291

申请日：2023-06-27

Applicant: NVIDIA Corporation

Inventor： Yilin Yang ,  Bala Siva Sashank Jujjavarapu ,  Pekka Janis ,  Zhaoting Ye ,  Sangmin Oh ,  Minwoo Park ,  Daniel Herrera Castro ,  Tommi Koivisto ,  David Nister

IPC: G06N3/08 ,  B60W30/14 ,  B60W60/00 ,  G06F18/214 ,  G06V10/762 ,  G06V20/56

CPC classification number: G06N3/08 ,  B60W30/14 ,  B60W60/0011 ,  G06F18/2155 ,  G06V10/763 ,  G06V20/56

Abstract: In various examples, a deep neural network (DNN) is trained to accurately predict, in deployment, distances to objects and obstacles using image data alone. The DNN may be trained with ground truth data that is generated and encoded using sensor data from any number of depth predicting sensors, such as, without limitation, RADAR sensors, LIDAR sensors, and/or SONAR sensors. Camera adaptation algorithms may be used in various embodiments to adapt the DNN for use with image data generated by cameras with varying parameters—such as varying fields of view. In some examples, a post-processing safety bounds operation may be executed on the predictions of the DNN to ensure that the predictions fall within a safety-permissible range.

公开(公告)号：US11960026B2

公开(公告)日：2024-04-16

申请号：US17976581

申请日：2022-10-28

Applicant: NVIDIA Corporation

Inventor： Alexander Popov ,  Nikolai Smolyanskiy ,  Ryan Oldja ,  Shane Murray ,  Tilman Wekel ,  David Nister ,  Joachim Pehserl ,  Ruchi Bhargava ,  Sangmin Oh

IPC: G01S7/41 ,  G01S13/86 ,  G01S13/89 ,  G06N3/04 ,  G06N3/08

CPC classification number: G01S7/417 ,  G01S13/865 ,  G01S13/89 ,  G06N3/04 ,  G06N3/08

Abstract: In various examples, a deep neural network(s) (e.g., a convolutional neural network) may be trained to detect moving and stationary obstacles from RADAR data of a three dimensional (3D) space. In some embodiments, ground truth training data for the neural network(s) may be generated from LIDAR data. More specifically, a scene may be observed with RADAR and LIDAR sensors to collect RADAR data and LIDAR data for a particular time slice. The RADAR data may be used for input training data, and the LIDAR data associated with the same or closest time slice as the RADAR data may be annotated with ground truth labels identifying objects to be detected. The LIDAR labels may be propagated to the RADAR data, and LIDAR labels containing less than some threshold number of RADAR detections may be omitted. The (remaining) LIDAR labels may be used to generate ground truth data.

公开(公告)号：US20230334317A1

公开(公告)日：2023-10-19

申请号：US18337854

申请日：2023-06-20

Applicant: NVIDIA Corporation

Inventor： Junghyun Kwon ,  Yilin Yang ,  Bala Siva Sashank Jujjavarapu ,  Zhaoting Ye ,  Sangmin Oh ,  Minwoo Park ,  David Nister

IPC: G06N3/08 ,  B60W30/14 ,  B60W60/00 ,  G06V20/56 ,  G06F18/214 ,  G06V10/762

CPC classification number: G06N3/08 ,  B60W30/14 ,  B60W60/0011 ,  G06V20/56 ,  G06F18/2155 ,  G06V10/763

Abstract: In various examples, a deep neural network (DNN) is trained—using image data alone—to accurately predict distances to objects, obstacles, and/or a detected free-space boundary. The DNN may be trained with ground truth data that is generated using sensor data representative of motion of an ego-vehicle and/or sensor data from any number of depth predicting sensors—such as, without limitation, RADAR sensors, LIDAR sensors, and/or SONAR sensors. The DNN may be trained using two or more loss functions each corresponding to a particular portion of the environment that depth is predicted for, such that—in deployment—more accurate depth estimates for objects, obstacles, and/or the detected free-space boundary are computed by the DNN.

公开(公告)号：US20230236314A1

公开(公告)日：2023-07-27

申请号：US17585141

申请日：2022-01-26

Applicant: NVIDIA Corporation

Inventor： Feng Jin ,  Nitin Bharadwaj ,  Shane Murray ,  James Hockridge Critchley ,  Sangmin Oh

IPC: G01S13/931 ,  G01S13/58 ,  G01S7/35

CPC classification number: G01S13/931 ,  G01S13/584 ,  G01S7/356

Abstract: In various examples, methods and systems are provided for sampling and transmitting the most useful information from a radar signal representing a scene while staying within the computational and storage confines of a standard automotive radar sensor and the bandwidth constraints of a standard communication link between a radar sensor and processing unit. Disclosed approaches may select a patch of frequency bins that correspond to radar signals based at least on proximities of the frequency bins to one or more frequency bins corresponding to at least one peak and/or detection point in the radar signals. Data representing samples corresponding to the patch of frequency bins may be transmitted to the processing unit and applied to one or more machine learning models in order to accurately classify, identify, and/or track objects.

公开(公告)号：US20230213945A1

公开(公告)日：2023-07-06

申请号：US17565837

申请日：2021-12-30

Applicant: NVIDIA Corporation

Inventor： Neeraj Sajjan ,  Mehmet K. Kocamaz ,  Junghyun Kwon ,  Sangmin Oh ,  Minwoo Park ,  David Nister

IPC: G05D1/02 ,  G06N3/08 ,  G05D1/00

CPC classification number: G05D1/0248 ,  G05D1/0257 ,  G06N3/08 ,  G05D1/0221 ,  G05D1/0219 ,  G05D1/0088 ,  G05D1/0251 ,  G05D1/0255 ,  G05D2201/0213

Abstract: In various examples, one or more output channels of a deep neural network (DNN) may be used to determine assignments of obstacles to paths. To increase the accuracy of the DNN, the input to the DNN may include an input image, one or more representations of path locations, and/or one or more representations of obstacle locations. The system may thus repurpose previously computed information—e.g., obstacle locations, path locations, etc.—from other operations of the system, and use them to generate more detailed inputs for the DNN to increase accuracy of the obstacle to path assignments. Once the output channels are computed using the DNN, computed bounding shapes for the objects may be compared to the outputs to determine the path assignments for each object.

公开(公告)号：US20220222477A1

公开(公告)日：2022-07-14

申请号：US17149437

申请日：2021-01-14

Applicant: NVIDIA Corporation

Inventor： Yichun Shen ,  Wanli Jiang ,  Junghyun Kwon ,  Siyi Li ,  Minwoo Park ,  Sangmin Oh

IPC: G06K9/00 ,  G06K9/62 ,  G06K9/20 ,  G06K9/46

Abstract: Apparatuses, systems, and techniques to perform non-maximum suppression (NMS) in parallel to remove redundant bounding boxes. In at least one embodiment, two or more parallel circuits to perform two or more portions of a NMS algorithm in parallel to remove one or more redundant bounding boxes corresponding to one or more objects within one or more digital images.

公开(公告)号：US20210026355A1

公开(公告)日：2021-01-28

申请号：US16938706

申请日：2020-07-24

Applicant: NVIDIA Corporation

Inventor： Ke Chen ,  Nikolai Smolyanskiy ,  Alexey Kamenev ,  Ryan Oldja ,  Tilman Wekel ,  David Nister ,  Joachim Pehserl ,  Ibrahim Eden ,  Sangmin Oh ,  Ruchi Bhargava

IPC: G05D1/00 ,  G06T7/10 ,  G06T5/50 ,  G06K9/00 ,  G06K9/62

Abstract: A deep neural network(s) (DNN) may be used to perform panoptic segmentation by performing pixel-level class and instance segmentation of a scene using a single pass of the DNN. Generally, one or more images and/or other sensor data may be stitched together, stacked, and/or combined, and fed into a DNN that includes a common trunk and several heads that predict different outputs. The DNN may include a class confidence head that predicts a confidence map representing pixels that belong to particular classes, an instance regression head that predicts object instance data for detected objects, an instance clustering head that predicts a confidence map of pixels that belong to particular instances, and/or a depth head that predicts range values. These outputs may be decoded to identify bounding shapes, class labels, instance labels, and/or range values for detected objects, and used to enable safe path planning and control of an autonomous vehicle.

公开(公告)号：US20200218979A1

公开(公告)日：2020-07-09

申请号：US16813306

申请日：2020-03-09

Applicant: NVIDIA Corporation

Inventor： Junghyun Kwon ,  Yilin Yang ,  Bala Siva Sashank Jujjavarapu ,  Zhaoting Ye ,  Sangmin Oh ,  Minwoo Park ,  David Nister

IPC: G06N3/08 ,  B60W30/14 ,  G06K9/00 ,  G06K9/62 ,  B60W60/00

Abstract: In various examples, a deep neural network (DNN) is trained—using image data alone—to accurately predict distances to objects, obstacles, and/or a detected free-space boundary. The DNN may be trained with ground truth data that is generated using sensor data representative of motion of an ego-vehicle and/or sensor data from any number of depth predicting sensors—such as, without limitation, RADAR sensors, LIDAR sensors, and/or SONAR sensors. The DNN may be trained using two or more loss functions each corresponding to a particular portion of the environment that depth is predicted for, such that—in deployment—more accurate depth estimates for objects, obstacles, and/or the detected free-space boundary are computed by the DNN. In some embodiments, a sampling algorithm may be used to sample depth values corresponding to an input resolution of the DNN from a predicted depth map of the DNN at an output resolution of the DNN.