公开(公告)号：US20240265712A1

公开(公告)日：2024-08-08

申请号：US18599719

申请日：2024-03-08

Applicant: NVIDIA Corporation

Inventor： David Wehr ,  Ibrahim Eden ,  Joachim Pehserl

IPC: G06V20/58 ,  G01B11/22 ,  G01S17/89 ,  G05D1/249 ,  G06N7/01 ,  G06T7/579 ,  G06T7/70

CPC classification number: G06V20/58 ,  G01B11/22 ,  G01S17/89 ,  G05D1/249 ,  G06N7/01 ,  G06T7/579 ,  G06T7/70 ,  G06T2207/10028 ,  G06T2207/20081 ,  G06T2207/30261

Abstract: In various examples, systems and methods are described that generate scene flow in 3D space through simplifying the 3D LiDAR data to “2.5D” optical flow space (e.g., x, y, and depth flow). For example, LiDAR range images may be used to generate 2.5D representations of depth flow information between frames of LiDAR data, and two or more range images may be compared to generate depth flow information, and messages may be passed—e.g., using a belief propagation algorithm—to update pixel values in the 2.5D representation. The resulting images may then be used to generate 2.5D motion vectors, and the 2.5D motion vectors may be converted back to 3D space to generate a 3D scene flow representation of an environment around an autonomous machine.

公开(公告)号：US20240029447A1

公开(公告)日：2024-01-25

申请号：US18482183

申请日：2023-10-06

Applicant: NVIDIA Corporation

Inventor： Nikolai SMOLYANSKIY ,  Ryan Oldja ,  Ke Chen ,  Alexander Popov ,  Joachim Pehserl ,  Ibrahim Eden ,  Tilman Wekel ,  David Wehr ,  Ruchi Bhargava ,  David Nister

IPC: G06V20/58 ,  G01S17/931 ,  B60W60/00 ,  G01S17/89 ,  G05D1/00 ,  G06T19/00 ,  G06N3/045

CPC classification number: G06V20/584 ,  G01S17/931 ,  B60W60/0016 ,  B60W60/0027 ,  B60W60/0011 ,  G01S17/89 ,  G05D1/0088 ,  G06T19/006 ,  G06V20/58 ,  G06N3/045 ,  B60W2420/403 ,  G06T2207/10028 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/30261

Abstract: A deep neural network(s) (DNN) may be used to detect objects from sensor data of a three dimensional (3D) environment. For example, a multi-view perception DNN may include multiple constituent DNNs or stages chained together that sequentially process different views of the 3D environment. An example DNN may include a first stage that performs class segmentation in a first view (e.g., perspective view) and a second stage that performs class segmentation and/or regresses instance geometry in a second view (e.g., top-down). The DNN outputs may be processed to generate 2D and/or 3D bounding boxes and class labels for detected objects in the 3D environment. As such, the techniques described herein may be used to detect and classify animate objects and/or parts of an environment, and these detections and classifications may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

公开(公告)号：US20230366698A1

公开(公告)日：2023-11-16

申请号：US18352578

申请日：2023-07-14

Applicant: NVIDIA Corporation

Inventor： David Nister ,  Ruchi Bhargava ,  Vaibhav Thukral ,  Michael Grabner ,  Ibrahim Eden ,  Jeffrey Liu

IPC: G01C21/00 ,  G01C21/16 ,  G06N3/02

CPC classification number: G01C21/3841 ,  G01C21/3896 ,  G01C21/3878 ,  G01C21/1652 ,  G01C21/3867 ,  G06N3/02 ,  G01C21/3811

Abstract: An end-to-end system for data generation, map creation using the generated data, and localization to the created map is disclosed. Mapstreams—or streams of sensor data, perception outputs from deep neural networks (DNNs), and/or relative trajectory data—corresponding to any number of drives by any number of vehicles may be generated and uploaded to the cloud. The mapstreams may be used to generate map data—and ultimately a fused high definition (HD) map—that represents data generated over a plurality of drives. When localizing to the fused HD map, individual localization results may be generated based on comparisons of real-time data from a sensor modality to map data corresponding to the same sensor modality. This process may be repeated for any number of sensor modalities and the results may be fused together to determine a final fused localization result.

公开(公告)号：US11532168B2

公开(公告)日：2022-12-20

申请号：US16915346

申请日：2020-06-29

Applicant: NVIDIA Corporation

Inventor： Nikolai Smolyanskiy ,  Ryan Oldja ,  Ke Chen ,  Alexander Popov ,  Joachim Pehserl ,  Ibrahim Eden ,  Tilman Wekel ,  David Wehr ,  Ruchi Bhargava ,  David Nister

IPC: G06V20/58 ,  G01S17/931 ,  B60W60/00 ,  G01S17/89 ,  G05D1/00 ,  G06N3/04 ,  G06T19/00

Abstract: A deep neural network(s) (DNN) may be used to detect objects from sensor data of a three dimensional (3D) environment. For example, a multi-view perception DNN may include multiple constituent DNNs or stages chained together that sequentially process different views of the 3D environment. An example DNN may include a first stage that performs class segmentation in a first view (e.g., perspective view) and a second stage that performs class segmentation and/or regresses instance geometry in a second view (e.g., top-down). The DNN outputs may be processed to generate 2D and/or 3D bounding boxes and class labels for detected objects in the 3D environment. As such, the techniques described herein may be used to detect and classify animate objects and/or parts of an environment, and these detections and classifications may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

公开(公告)号：US20190266736A1

公开(公告)日：2019-08-29

申请号：US16051263

申请日：2018-07-31

Applicant: Nvidia Corporation

Inventor： Ishwar Kulkarni ,  Ibrahim Eden ,  Michael Kroepfl ,  David Nister

IPC: G06T7/30 ,  G06T15/04 ,  G01S17/89 ,  G06K9/00

Abstract: Various types of systems or technologies can be used to collect data in a 3D space. For example, LiDAR (light detection and ranging) and RADAR (radio detection and ranging) systems are commonly used to generate point cloud data for 3D space around vehicles, for such functions as localization, mapping, and tracking. This disclosure provides improvements for processing the point cloud data that has been collected. The processing improvements include analyzing point cloud data using trajectory equations, depth maps, and texture maps. The processing improvements also include representing the point cloud data by a two dimensional depth map or a texture map and using the depth map or texture map to provide object motion, obstacle detection, freespace detection, and landmark detection for an area surrounding a vehicle.

公开(公告)号：US11954914B2

公开(公告)日：2024-04-09

申请号：US17392050

申请日：2021-08-02

Applicant: NVIDIA Corporation

Inventor： David Wehr ,  Ibrahim Eden ,  Joachim Pehserl

IPC: G06V20/58 ,  G01B11/22 ,  G01S17/89 ,  G05D1/00 ,  G05D1/02 ,  G06N7/01 ,  G06T7/579 ,  G06T7/70

CPC classification number: G06V20/58 ,  G01B11/22 ,  G01S17/89 ,  G05D1/0231 ,  G06N7/01 ,  G06T7/579 ,  G06T7/70 ,  G06T2207/10028 ,  G06T2207/20081 ,  G06T2207/30261

Abstract: In various examples, systems and methods are described that generate scene flow in 3D space through simplifying the 3D LiDAR data to “2.5D” optical flow space (e.g., x, y, and depth flow). For example, LiDAR range images may be used to generate 2.5D representations of depth flow information between frames of LiDAR data, and two or more range images may be compared to generate depth flow information, and messages may be passed—e.g., using a belief propagation algorithm—to update pixel values in the 2.5D representation. The resulting images may then be used to generate 2.5D motion vectors, and the 2.5D motion vectors may be converted back to 3D space to generate a 3D scene flow representation of an environment around an autonomous machine.

公开(公告)号：US11906660B2

公开(公告)日：2024-02-20

申请号：US17005788

申请日：2020-08-28

Applicant: NVIDIA Corporation

Inventor： Tilman Wekel ,  Sangmin Oh ,  David Nister ,  Joachim Pehserl ,  Neda Cvijetic ,  Ibrahim Eden

IPC: G01S7/00 ,  G01S7/48 ,  G01S17/894 ,  G01S7/481 ,  G01S17/931 ,  G06V10/764 ,  G06V10/80 ,  G06V10/82 ,  G06V20/58 ,  G01S7/28

CPC classification number: G01S7/4802 ,  G01S7/481 ,  G01S17/894 ,  G01S17/931 ,  G06V10/764 ,  G06V10/80 ,  G06V10/82 ,  G06V20/58 ,  G01S7/28

Abstract: In various examples, a deep neural network (DNN) may be used to detect and classify animate objects and/or parts of an environment. The DNN may be trained using camera-to-LiDAR cross injection to generate reliable ground truth data for LiDAR range images. For example, annotations generated in the image domain may be propagated to the LiDAR domain to increase the accuracy of the ground truth data in the LiDAR domain—e.g., without requiring manual annotation in the LiDAR domain. Once trained, the DNN may output instance segmentation masks, class segmentation masks, and/or bounding shape proposals corresponding to two-dimensional (2D) LiDAR range images, and the outputs may be fused together to project the outputs into three-dimensional (3D) LiDAR point clouds. This 2D and/or 3D information output by the DNN may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

公开(公告)号：US11842440B2

公开(公告)日：2023-12-12

申请号：US17228460

申请日：2021-04-12

Applicant: NVIDIA Corporation

Inventor： Philippe Bouttefroy ,  David Nister ,  Ibrahim Eden

IPC: G06T17/05 ,  G06V20/56 ,  G06V10/764 ,  G06V10/82

CPC classification number: G06T17/05 ,  G06V10/764 ,  G06V10/82 ,  G06V20/56

Abstract: In various examples, locations of directional landmarks, such as vertical landmarks, may be identified using 3D reconstruction. A set of observations of directional landmarks (e.g., images captured from a moving vehicle) may be reduced to 1D lookups by rectifying the observations to align directional landmarks along a particular direction of the observations. Object detection may be applied, and corresponding 1D lookups may be generated to represent the presence of a detected vertical landmark in an image.

公开(公告)号：US11698272B2

公开(公告)日：2023-07-11

申请号：US17007873

申请日：2020-08-31

Applicant: NVIDIA Corporation

Inventor： Michael Kroepfl ,  Amir Akbarzadeh ,  Ruchi Bhargava ,  Vaibhav Thukral ,  Neda Cvijetic ,  Vadim Cugunovs ,  David Nister ,  Birgit Henke ,  Ibrahim Eden ,  Youding Zhu ,  Michael Grabner ,  Ivana Stojanovic ,  Yu Sheng ,  Jeffrey Liu ,  Enliang Zheng ,  Jordan Marr ,  Andrew Carley

IPC: G01C21/00 ,  G06N3/02 ,  G01C21/16

CPC classification number: G01C21/3841 ,  G01C21/1652 ,  G01C21/3811 ,  G01C21/3867 ,  G01C21/3878 ,  G01C21/3896 ,  G06N3/02

Abstract: An end-to-end system for data generation, map creation using the generated data, and localization to the created map is disclosed. Mapstreams—or streams of sensor data, perception outputs from deep neural networks (DNNs), and/or relative trajectory data—corresponding to any number of drives by any number of vehicles may be generated and uploaded to the cloud. The mapstreams may be used to generate map data—and ultimately a fused high definition (HD) map—that represents data generated over a plurality of drives. When localizing to the fused HD map, individual localization results may be generated based on comparisons of real-time data from a sensor modality to map data corresponding to the same sensor modality. This process may be repeated for any number of sensor modalities and the results may be fused together to determine a final fused localization result.

公开(公告)号：US20230033470A1

公开(公告)日：2023-02-02

申请号：US17392050

申请日：2021-08-02

Applicant: NVIDIA Corporation

Inventor： David Wehr ,  Ibrahim Eden ,  Joachim Pehserl

IPC: G06K9/00 ,  G06T7/579 ,  G06T7/70 ,  G05D1/02 ,  G06N7/00 ,  G01B11/22 ,  G01S17/89

Abstract: In various examples, systems and methods are described that generate scene flow in 3D space through simplifying the 3D LiDAR data to “2.5D” optical flow space (e.g., x, y, and depth flow). For example, LiDAR range images may be used to generate 2.5D representations of depth flow information between frames of LiDAR data, and two or more range images may be compared to generate depth flow information, and messages may be passed—e.g., using a belief propagation algorithm—to update pixel values in the 2.5D representation. The resulting images may then be used to generate 2.5D motion vectors, and the 2.5D motion vectors may be converted back to 3D space to generate a 3D scene flow representation of an environment around an autonomous machine.