公开(公告)号：US20240312187A1

公开(公告)日：2024-09-19

申请号：US18184071

申请日：2023-03-15

Applicant: NVIDIA Corporation

Inventor： Yue Wu ,  Cheng-Chieh Yang ,  Xin Tong ,  Minwoo Park

IPC: G06V10/771 ,  G06V10/77

CPC classification number: G06V10/771 ,  G06V10/7715

Abstract: In various examples, feature tracking for autonomous or semi-autonomous systems and applications is described herein. Systems and methods are disclosed that merge, using one or more processes, features detected using a feature tracker(s) and features detected using a feature detector(s) in order to track features between images. In some examples, the number of merged features and/or the locations of the merged features within the images are limited. This way, the systems and methods are able to identify merged features that are of greater importance for tracking while refraining from tracking merged features that are of less importance. For example, if the systems and methods are being used to identify features for autonomous driving, a greater number of merged features that are associated with objects located proximate to the driving surface may be tracked as compared to merged features that are associated with the sky.

公开(公告)号：US20240273926A1

公开(公告)日：2024-08-15

申请号：US18647261

申请日：2024-04-26

Applicant: NVIDIA CORPORATION

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: G06V20/64 ,  B60G17/0165 ,  B60K31/00 ,  B60W60/00 ,  G01S17/89 ,  G01S17/931 ,  G06F18/214 ,  G06V20/58

CPC classification number: G06V20/64 ,  G01S17/89 ,  G01S17/931 ,  G06F18/214 ,  G06V20/58 ,  B60G17/0165 ,  B60K31/00 ,  B60W60/001 ,  B60W2420/408

Abstract: In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated from real-world data. For example, one or more vehicles may collect image data and LiDAR data while navigating through a real-world environment. To generate input training data, 3D surface structure estimation may be performed on captured image data to generate a sparse representation of a 3D surface structure of interest (e.g., a 3D road surface). To generate corresponding ground truth training data, captured LiDAR data may be smoothed, subject to outlier removal, subject to triangulation to filling missing values, accumulated from multiple LiDAR sensors, aligned with corresponding frames of image data, and/or annotated to identify 3D points on the 3D surface of interest, and the identified 3D points may be projected to generate a dense representation of the 3D surface structure.

公开(公告)号：US12008822B2

公开(公告)日：2024-06-11

申请号：US17452752

申请日：2021-10-28

Applicant: NVIDIA Corporation

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: G06V20/64 ,  G01S17/89 ,  G01S17/931 ,  G06F18/214 ,  G06V20/58 ,  B60G17/0165 ,  B60K31/00 ,  B60W60/00

CPC classification number: G06V20/64 ,  G01S17/89 ,  G01S17/931 ,  G06F18/214 ,  G06V20/58 ,  B60G17/0165 ,  B60K31/00 ,  B60W60/001 ,  B60W2420/408

Abstract: In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated from real-world data. For example, one or more vehicles may collect image data and LiDAR data while navigating through a real-world environment. To generate input training data, 3D surface structure estimation may be performed on captured image data to generate a sparse representation of a 3D surface structure of interest (e.g., a 3D road surface). To generate corresponding ground truth training data, captured LiDAR data may be smoothed, subject to outlier removal, subject to triangulation to filling missing values, accumulated from multiple LiDAR sensors, aligned with corresponding frames of image data, and/or annotated to identify 3D points on the 3D surface of interest, and the identified 3D points may be projected to generate a dense representation of the 3D surface structure.

公开(公告)号：US20240161341A1

公开(公告)日：2024-05-16

申请号：US18166118

申请日：2023-02-08

Applicant: NVIDIA Corporation

Inventor： Ayon Sen ,  Gang Pan ,  Cheng-Chieh Yang ,  Yue Wu

IPC: G06T7/80

CPC classification number: G06T7/80 ,  G06T2207/10028 ,  G06T2207/20084

Abstract: In various examples, sensor configuration for autonomous or semi-autonomous systems and applications is described. Systems and methods are disclosed that may use image feature correspondences between camera images along with an assumption that image features are locally planar to determine parameters for calibrating an image sensor with a LiDAR sensor and/or another image sensor. In some examples, an optimization problem is constructed that attempts to minimize a geometric loss function, where the geometric loss function encodes the notion that corresponding image features are views of a same point on a locally planar surface (e.g., a surfel or mesh) that is constructed from LiDAR data generated using a LiDAR sensor. In some examples, performing such processes to determine the calibration parameters may remove structure estimation from the optimization problem.

公开(公告)号：US11967022B2

公开(公告)日：2024-04-23

申请号：US17452751

申请日：2021-10-28

Applicant: NVIDIA Corporation

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: G06T17/05 ,  B60W30/09 ,  B60W30/14 ,  B60W40/06 ,  B60W50/06 ,  B60W60/00 ,  G06F18/214 ,  G06V20/05 ,  G06V20/58

CPC classification number: G06T17/05 ,  B60W30/09 ,  B60W30/143 ,  B60W40/06 ,  B60W50/06 ,  B60W60/001 ,  G06F18/214 ,  G06V20/05 ,  B60W2420/42 ,  B60W2420/52 ,  B60W2552/15

Abstract: In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated using a parametric mathematical modeling. A variety of synthetic 3D road surfaces may be generated by modeling a 3D road surface using varied parameters to simulate changes in road direction and lateral surface slope. In an example embodiment, a synthetic 3D road surface may be created by modeling a longitudinal 3D curve and expanding the longitudinal 3D curve to a 3D surface, and the resulting synthetic 3D surface may be sampled to form a synthetic ground truth projection image (e.g., a 2D height map). To generate corresponding input training data, a known pattern that represents which pixels may remain unobserved during 3D structure estimation may be generated and applied to a ground truth projection image to simulate a corresponding sparse projection image.

公开(公告)号：US20240062657A1

公开(公告)日：2024-02-22

申请号：US18491492

申请日：2023-10-20

Applicant: NVIDIA Corporation

Inventor： Yue Wu ,  Pekka Janis ,  Xin Tong ,  Cheng-Chieh Yang ,  Minwoo Park ,  David Nister

IPC: G08G1/16 ,  G06V10/82 ,  G06V20/58 ,  G06V20/10 ,  G06F18/214 ,  G05D1/00 ,  G05D1/02 ,  G06N3/04 ,  G06T7/20

CPC classification number: G08G1/166 ,  G06V10/82 ,  G06V20/58 ,  G06V20/10 ,  G06F18/214 ,  G05D1/0088 ,  G05D1/0289 ,  G06N3/0418 ,  G06T7/20 ,  G05D2201/0213

Abstract: In various examples, a sequential deep neural network (DNN) may be trained using ground truth data generated by correlating (e.g., by cross-sensor fusion) sensor data with image data representative of a sequences of images. In deployment, the sequential DNN may leverage the sensor correlation to compute various predictions using image data alone. The predictions may include velocities, in world space, of objects in fields of view of an ego-vehicle, current and future locations of the objects in image space, and/or a time-to-collision (TTC) between the objects and the ego-vehicle. These predictions may be used as part of a perception system for understanding and reacting to a current physical environment of the ego-vehicle.

公开(公告)号：US20240059295A1

公开(公告)日：2024-02-22

申请号：US18498370

申请日：2023-10-31

Applicant: NVIDIA Corporation

Inventor： Minwoo Park ,  Yue Wu ,  Cheng-Chieh Yang

IPC: B60W40/02 ,  G06T7/30 ,  G06T7/80 ,  H04N13/296 ,  H04N13/271 ,  G06T15/10 ,  G06V20/58 ,  B60W60/00 ,  G06T7/593

CPC classification number: B60W40/02 ,  G06T7/30 ,  G06T7/85 ,  H04N13/296 ,  H04N13/271 ,  G06T15/10 ,  G06V20/58 ,  B60W60/001 ,  G06T7/593 ,  B60W2420/42 ,  G06T2207/10012 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/20132 ,  G06T2207/20228 ,  G06T2207/30261 ,  G06T2207/30244

Abstract: In various examples, systems and methods are disclosed that detect hazards on a roadway by identifying discontinuities between pixels on a depth map. For example, two synchronized stereo cameras mounted on an ego-machine may generate images that may be used extract depth or disparity information. Because a hazard's height may cause an occlusion of the driving surface behind the hazard from a perspective of a camera(s), a discontinuity in disparity values may indicate the presence of a hazard. For example, the system may analyze pairs of pixels on the depth map and, when the system determines that a disparity between a pair of pixels satisfies a disparity threshold, the system may identify the pixel nearest the ego-machine as a hazard pixel.

公开(公告)号：US11657532B2

公开(公告)日：2023-05-23

申请号：US17103680

申请日：2020-11-24

Applicant: NVIDIA Corporation

Inventor： Minwoo Park ,  Yue Wu ,  Michael Grabner ,  Cheng-Chieh Yang

IPC: G06T7/60 ,  G06T7/579 ,  G06V20/56

CPC classification number: G06T7/60 ,  G06T7/579 ,  G06V20/588 ,  G06T2200/08 ,  G06T2207/10028 ,  G06T2207/30256

Abstract: In various examples, surface profile estimation and bump detection may be performed based on a three-dimensional (3D) point cloud. The 3D point cloud may be filtered in view of a portion of an environment including drivable free-space, and within a threshold height to factor out other objects or obstacles other than a driving surface and protuberances thereon. The 3D point cloud may be analyzed—e.g., using a sliding window of bounding shapes along a longitudinal or other heading direction—to determine one-dimensional (1D) signal profiles corresponding to heights along the driving surface. The profile itself may be used by a vehicle—e.g., an autonomous or semi-autonomous vehicle—to help in navigating the environment, and/or the profile may be used to detect bumps, humps, and/or other protuberances along the driving surface, in addition to a location, orientation, and geometry thereof.

公开(公告)号：US20220188608A1

公开(公告)日：2022-06-16

申请号：US17122598

申请日：2020-12-15

Applicant: NVIDIA Corporation

Inventor： Gregory Heinrich ,  Maxim Milakov ,  Xin Tong ,  Yue Wu

IPC: G06N3/063 ,  G06N5/04 ,  G06F12/0893

Abstract: Apparatuses, systems, and techniques to cache and reuse data for a neural network. In at least one embodiment, data generated by one or more layers of a neural network is cached and reused by the neural network.