公开(公告)号：US20230135234A1

公开(公告)日：2023-05-04

申请号：US17452752

申请日：2021-10-28

Applicant: NVIDIA Corporation

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: G06K9/00 ,  G06K9/62 ,  G01S17/931 ,  G01S17/89

Abstract: In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated from real-world data. For example, one or more vehicles may collect image data and LiDAR data while navigating through a real-world environment. To generate input training data, 3D surface structure estimation may be performed on captured image data to generate a sparse representation of a 3D surface structure of interest (e.g., a 3D road surface). To generate corresponding ground truth training data, captured LiDAR data may be smoothed, subject to outlier removal, subject to triangulation to filling missing values, accumulated from multiple LiDAR sensors, aligned with corresponding frames of image data, and/or annotated to identify 3D points on the 3D surface of interest, and the identified 3D points may be projected to generate a dense representation of the 3D surface structure.

公开(公告)号：US20250022175A1

公开(公告)日：2025-01-16

申请号：US18349779

申请日：2023-07-10

Applicant: NVIDIA Corporation

Inventor： Yue Wu ,  Cheng-Chieh Yang ,  Kang Wang ,  Ayon Sen ,  Hsin Miao

IPC: G06T7/80 ,  G06T7/73 ,  H04N13/246

Abstract: In various examples, sensor calibration for autonomous or semi-autonomous systems and applications is described herein. Systems and methods are disclosed that calibrate image sensors, such as cameras, using images captured by the image sensors at different time instances. For instance, a first image sensor may generate first image data representing at least two images and a second image sensor may generate second image data representing at least one image. One or more feature points may then be tracked between the images represented by the first image data and the image represented by the second image data. Additionally, the feature point(s), timestamps associated with the images, poses associated with image sensors (e.g., poses of a vehicle), and/or other information may be used to determine one or more values of one or more parameters that calibrate the first image sensor with the second image sensor.

公开(公告)号：US12190448B2

公开(公告)日：2025-01-07

申请号：US17452749

申请日：2021-10-28

Applicant: NVIDIA Corporation

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: G06T17/20 ,  B60W30/14 ,  B60W40/06 ,  B60W50/06 ,  B60W60/00 ,  G06F18/214 ,  G06F18/24 ,  G06N3/08 ,  G06T7/11 ,  G06T7/40

Abstract: In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated using a simulated environment. For example, a simulation may be run to simulate a virtual world or environment, render frames of virtual sensor data (e.g., images), and generate corresponding depth maps and segmentation masks (identifying a component of the simulated environment such as a road). To generate input training data, 3D structure estimation may be performed on a rendered frame to generate a representation of a 3D surface structure of the road. To generate corresponding ground truth training data, a corresponding depth map and segmentation mask may be used to generate a dense representation of the 3D surface structure.

公开(公告)号：US20240273926A1

公开(公告)日：2024-08-15

申请号：US18647261

申请日：2024-04-26

Applicant: NVIDIA CORPORATION

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: G06V20/64 ,  B60G17/0165 ,  B60K31/00 ,  B60W60/00 ,  G01S17/89 ,  G01S17/931 ,  G06F18/214 ,  G06V20/58

CPC classification number: G06V20/64 ,  G01S17/89 ,  G01S17/931 ,  G06F18/214 ,  G06V20/58 ,  B60G17/0165 ,  B60K31/00 ,  B60W60/001 ,  B60W2420/408

Abstract: In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated from real-world data. For example, one or more vehicles may collect image data and LiDAR data while navigating through a real-world environment. To generate input training data, 3D surface structure estimation may be performed on captured image data to generate a sparse representation of a 3D surface structure of interest (e.g., a 3D road surface). To generate corresponding ground truth training data, captured LiDAR data may be smoothed, subject to outlier removal, subject to triangulation to filling missing values, accumulated from multiple LiDAR sensors, aligned with corresponding frames of image data, and/or annotated to identify 3D points on the 3D surface of interest, and the identified 3D points may be projected to generate a dense representation of the 3D surface structure.

公开(公告)号：US12008822B2

公开(公告)日：2024-06-11

申请号：US17452752

申请日：2021-10-28

Applicant: NVIDIA Corporation

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: G06V20/64 ,  G01S17/89 ,  G01S17/931 ,  G06F18/214 ,  G06V20/58 ,  B60G17/0165 ,  B60K31/00 ,  B60W60/00

CPC classification number: G06V20/64 ,  G01S17/89 ,  G01S17/931 ,  G06F18/214 ,  G06V20/58 ,  B60G17/0165 ,  B60K31/00 ,  B60W60/001 ,  B60W2420/408

Abstract: In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated from real-world data. For example, one or more vehicles may collect image data and LiDAR data while navigating through a real-world environment. To generate input training data, 3D surface structure estimation may be performed on captured image data to generate a sparse representation of a 3D surface structure of interest (e.g., a 3D road surface). To generate corresponding ground truth training data, captured LiDAR data may be smoothed, subject to outlier removal, subject to triangulation to filling missing values, accumulated from multiple LiDAR sensors, aligned with corresponding frames of image data, and/or annotated to identify 3D points on the 3D surface of interest, and the identified 3D points may be projected to generate a dense representation of the 3D surface structure.

公开(公告)号：US11967022B2

公开(公告)日：2024-04-23

申请号：US17452751

申请日：2021-10-28

Applicant: NVIDIA Corporation

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: G06T17/05 ,  B60W30/09 ,  B60W30/14 ,  B60W40/06 ,  B60W50/06 ,  B60W60/00 ,  G06F18/214 ,  G06V20/05 ,  G06V20/58

CPC classification number: G06T17/05 ,  B60W30/09 ,  B60W30/143 ,  B60W40/06 ,  B60W50/06 ,  B60W60/001 ,  G06F18/214 ,  G06V20/05 ,  B60W2420/42 ,  B60W2420/52 ,  B60W2552/15

Abstract: In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated using a parametric mathematical modeling. A variety of synthetic 3D road surfaces may be generated by modeling a 3D road surface using varied parameters to simulate changes in road direction and lateral surface slope. In an example embodiment, a synthetic 3D road surface may be created by modeling a longitudinal 3D curve and expanding the longitudinal 3D curve to a 3D surface, and the resulting synthetic 3D surface may be sampled to form a synthetic ground truth projection image (e.g., a 2D height map). To generate corresponding input training data, a known pattern that represents which pixels may remain unobserved during 3D structure estimation may be generated and applied to a ground truth projection image to simulate a corresponding sparse projection image.