公开(公告)号：US12172667B2

公开(公告)日：2024-12-24

申请号：US17452747

申请日：2021-10-28

Applicant: NVIDIA Corporation

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: B60W60/00 ,  B60W40/09 ,  G06N3/02

Abstract: In various examples, a 3D surface structure such as the 3D surface structure of a road (3D road surface) may be observed and estimated to generate a 3D point cloud or other representation of the 3D surface structure. Since the estimated representation may be sparse, a deep neural network (DNN) may be used to predict values for a dense representation of the 3D surface structure from the sparse representation. For example, a sparse 3D point cloud may be projected to form a sparse projection image (e.g., a sparse 2D height map), which may be fed into the DNN to predict a dense projection image (e.g., a dense 2D height map). The predicted dense representation of the 3D surface structure may be provided to an autonomous vehicle drive stack to enable safe and comfortable planning and control of the autonomous vehicle.

公开(公告)号：US12100230B2

公开(公告)日：2024-09-24

申请号：US17452752

申请日：2021-10-28

Applicant: NVIDIA Corporation

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: G06V20/64 ,  G01S17/89 ,  G01S17/931 ,  G06F18/214 ,  G06V20/58 ,  B60G17/0165 ,  B60K31/00 ,  B60W60/00

CPC classification number: G06V20/64 ,  G01S17/89 ,  G01S17/931 ,  G06F18/214 ,  G06V20/58 ,  B60G17/0165 ,  B60K31/00 ,  B60W60/001 ,  B60W2420/408

Abstract: In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated from real-world data. For example, one or more vehicles may collect image data and LiDAR data while navigating through a real-world environment. To generate input training data, 3D surface structure estimation may be performed on captured image data to generate a sparse representation of a 3D surface structure of interest (e.g., a 3D road surface). To generate corresponding ground truth training data, captured LiDAR data may be smoothed, subject to outlier removal, subject to triangulation to filling missing values, accumulated from multiple LiDAR sensors, aligned with corresponding frames of image data, and/or annotated to identify 3D points on the 3D surface of interest, and the identified 3D points may be projected to generate a dense representation of the 3D surface structure.

公开(公告)号：US20230135088A1

公开(公告)日：2023-05-04

申请号：US17452747

申请日：2021-10-28

Applicant: NVIDIA Corporation

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: B60W60/00 ,  B60W40/09 ,  G06N3/02

Abstract: In various examples, a 3D surface structure such as the 3D surface structure of a road (3D road surface) may be observed and estimated to generate a 3D point cloud or other representation of the 3D surface structure. Since the estimated representation may be sparse, a deep neural network (DNN) may be used to predict values for a dense representation of the 3D surface structure from the sparse representation. For example, a sparse 3D point cloud may be projected to form a sparse projection image (e.g., a sparse 2D height map), which may be fed into the DNN to predict a dense projection image (a dense 21) height map). The predicted dense representation of the 3D surface structure may be provided to an autonomous vehicle drive stack to enable safe and comfortable planning and control of the autonomous vehicle.

公开(公告)号：US12039663B2

公开(公告)日：2024-07-16

申请号：US17452751

申请日：2021-10-28

Applicant: NVIDIA Corporation

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: G06T17/05 ,  B60W30/09 ,  B60W30/14 ,  B60W40/06 ,  B60W50/06 ,  B60W60/00 ,  G06F18/214 ,  G06V20/05 ,  G06V20/58

CPC classification number: G06T17/05 ,  B60W30/09 ,  B60W30/143 ,  B60W40/06 ,  B60W50/06 ,  B60W60/001 ,  G06F18/214 ,  G06V20/05 ,  B60W2420/42 ,  B60W2420/52 ,  B60W2552/15

Abstract: In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated using a parametric mathematical modeling. A variety of synthetic 3D road surfaces may be generated by modeling a 3D road surface using varied parameters to simulate changes in road direction and lateral surface slope. In an example embodiment, a synthetic 3D road surface may be created by modeling a longitudinal 3D curve and expanding the longitudinal 3D curve to a 3D surface, and the resulting synthetic 3D surface may be sampled to form a synthetic ground truth projection image (e.g., a 2D height map). To generate corresponding input training data, a known pattern that represents which pixels may remain unobserved during 3D structure estimation may be generated and applied to a ground truth projection image to simulate a corresponding sparse projection image.

公开(公告)号：US20240161342A1

公开(公告)日：2024-05-16

申请号：US18166121

申请日：2023-02-08

Applicant: NVIDIA Corporation

Inventor： Ayon Sen ,  Gang Pan ,  Cheng-Chieh Yang ,  Yue Wu

IPC: G06T7/80 ,  G01S17/86 ,  G01S17/89 ,  G01S17/931 ,  H04N17/00

CPC classification number: G06T7/80 ,  G01S17/86 ,  G01S17/89 ,  G01S17/931 ,  H04N17/002 ,  G06T2207/10028 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/30244

Abstract: In various examples, sensor configuration for autonomous or semi-autonomous systems and applications is described. Systems and methods are disclosed that may use image feature correspondences between camera images along with an assumption that image features are locally planar to determine parameters for calibrating an image sensor with a LiDAR sensor and/or another image sensor. In some examples, an optimization problem is constructed that attempts to minimize a geometric loss function, where the geometric loss function encodes the notion that corresponding image features are views of a same point on a locally planar surface (e.g., a surfel or mesh) that is constructed from LiDAR data generated using a LiDAR sensor. In some examples, performing such processes to determine the calibration parameters may remove structure estimation from the optimization problem.

公开(公告)号：US20230351769A1

公开(公告)日：2023-11-02

申请号：US17733508

申请日：2022-04-29

Applicant: NVIDIA Corporation

Inventor： Yue WU ,  Liwen Lin ,  Xin Tong ,  Gang Pan

IPC: G06V20/58 ,  G06T7/55 ,  G06V10/74 ,  G06V10/764 ,  G06V10/762

CPC classification number: G06V20/58 ,  G06T7/55 ,  G06V10/761 ,  G06V10/764 ,  G06V10/762 ,  G06T2207/20081 ,  G06T2207/30261 ,  B60W60/0015

Abstract: In various examples, systems and methods for machine learning based hazard detection for autonomous machine applications using stereo disparity are presented. Disparity between a stereo pair of images is used to generate a path disparity model. Using the path disparity model, a machine learning model can recognize when a pixel in the first image corresponds to a pixel in the second image even though the pixel in the two images does not have identical characteristics. Similarities in extracted feature vectors can be computed and represented by a vector similarity metric that is input to a machine learning classifier, along with feature information extracted from the stereo image pair, to differentiate hazard pixels from non-hazard pixels. In some embodiments, a V-space disparity map, where a first axis corresponds to disparity values and the second axis corresponds to pixel rows, may be used to simplify estimation of the path disparity model.

公开(公告)号：US20230351638A1

公开(公告)日：2023-11-02

申请号：US17733497

申请日：2022-04-29

Applicant: NVIDIA Corporation

Inventor： Yue WU ,  Liwen Lin ,  Cheng-Chieh Yang ,  Gang Pan

IPC: G06T7/00 ,  G06V20/56 ,  G06V10/762 ,  G06V10/764 ,  G06V10/75 ,  G06V10/25

CPC classification number: G06T7/97 ,  G06V20/56 ,  G06V10/762 ,  G06V10/764 ,  G06V10/751 ,  G06V10/25 ,  G06T2207/10012 ,  G06T2207/20021 ,  G06T2207/20228 ,  G06T2207/30252

Abstract: In various examples, system and methods for stereo disparity based hazard detection for autonomous machine applications are presented. Example embodiments may assist an ego-machine in detecting hazards within its path of travel. The systems and methods may use disparity between a stereo pair of images to generate a baseline path disparity model and further identify hazards from detected disparities that deviate from that path disparity model. A disparity map for the image pair is constructed in which each pixel represents a disparity for a corresponding element of the image captured. Blockwise division may be optionally used to subdivide the disparity map into a plurality of smaller disparity maps, each corresponding to a block of pixels of the disparity map. A V-space disparity map, where a first axis corresponds to disparity values and the second axis corresponds to pixel rows, may be used to simplify estimation of the path disparity model.

公开(公告)号：US20230136860A1

公开(公告)日：2023-05-04

申请号：US17452751

申请日：2021-10-28

Applicant: NVIDIA Corporation

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: G06T17/05 ,  G06K9/62 ,  G06K9/00 ,  B60W60/00 ,  B60W40/06 ,  B60W30/14 ,  B60W30/09 ,  B60W50/06

Abstract: In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated using a parametric mathematical modeling. A variety of synthetic 3D road surfaces may be generated by modeling a 3D road surface using varied parameters to simulate changes in road direction and lateral surface slope. In an example embodiment, a synthetic 3D road surface may be created by modeling a longitudinal 3D curve and expanding the longitudinal 3D curve to a 3D surface, and the resulting synthetic 3D surface may be sampled to form a synthetic ground truth projection image (e.g., a 2D height map). To generate corresponding input training data, a known pattern that represents which pixels may remain unobserved during 3D structure estimation may be generated and applied to a ground truth projection image to simulate a corresponding sparse projection image.

公开(公告)号：US20230136235A1

公开(公告)日：2023-05-04

申请号：US17452744

申请日：2021-10-28

Applicant: NVIDIA Corporation

Inventor： Kang Wang ,  Yue Wu ,  Minwoo Park ,  Gang Pan

IPC: B60W60/00 ,  B60W40/06 ,  G06K9/00 ,  B60W40/105

Abstract: In various examples, a 3D surface structure such as the 3D surface structure of a road (3D road surface) may be observed and estimated to generate a 3D point cloud or other representation of the 3D surface structure. Since the representation may be sparse, one or more densification techniques may be applied to densify the representation of the 3D surface structure. For example, the relationship between sparse and dense projection images (e.g., 2D height maps) may be modeled with a Markov random field, and Maximum a Posterior (MAP) inference may be performed using a corresponding joint probability distribution to estimate the most likely dense values given the sparse values. The resulting dense representation of the 3D surface structure may be provided to an autonomous vehicle drive stack to enable safe and comfortable planning and control of the autonomous vehicle.

公开(公告)号：US12288363B2

公开(公告)日：2025-04-29

申请号：US18166118

申请日：2023-02-08

Applicant: NVIDIA Corporation

Inventor： Ayon Sen ,  Gang Pan ,  Cheng-Chieh Yang ,  Yue Wu

IPC: G06T7/80 ,  G01S17/86 ,  G01S17/89 ,  G01S17/931 ,  H04N17/00

Abstract: In various examples, sensor configuration for autonomous or semi-autonomous systems and applications is described. Systems and methods are disclosed that may use image feature correspondences between camera images along with an assumption that image features are locally planar to determine parameters for calibrating an image sensor with a LiDAR sensor and/or another image sensor. In some examples, an optimization problem is constructed that attempts to minimize a geometric loss function, where the geometric loss function encodes the notion that corresponding image features are views of a same point on a locally planar surface (e.g., a surfel or mesh) that is constructed from LiDAR data generated using a LiDAR sensor. In some examples, performing such processes to determine the calibration parameters may remove structure estimation from the optimization problem.