公开(公告)号：US11532168B2

公开(公告)日：2022-12-20

申请号：US16915346

申请日：2020-06-29

Applicant: NVIDIA Corporation

Inventor： Nikolai Smolyanskiy ,  Ryan Oldja ,  Ke Chen ,  Alexander Popov ,  Joachim Pehserl ,  Ibrahim Eden ,  Tilman Wekel ,  David Wehr ,  Ruchi Bhargava ,  David Nister

IPC: G06V20/58 ,  G01S17/931 ,  B60W60/00 ,  G01S17/89 ,  G05D1/00 ,  G06N3/04 ,  G06T19/00

Abstract: A deep neural network(s) (DNN) may be used to detect objects from sensor data of a three dimensional (3D) environment. For example, a multi-view perception DNN may include multiple constituent DNNs or stages chained together that sequentially process different views of the 3D environment. An example DNN may include a first stage that performs class segmentation in a first view (e.g., perspective view) and a second stage that performs class segmentation and/or regresses instance geometry in a second view (e.g., top-down). The DNN outputs may be processed to generate 2D and/or 3D bounding boxes and class labels for detected objects in the 3D environment. As such, the techniques described herein may be used to detect and classify animate objects and/or parts of an environment, and these detections and classifications may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

公开(公告)号：US20210342608A1

公开(公告)日：2021-11-04

申请号：US17377053

申请日：2021-07-15

Applicant: NVIDIA Corporation

Inventor： Nikolai Smolyanskiy ,  Ryan Oldja ,  Ke Chen ,  Alexander Popov ,  Joachim Pehserl ,  Ibrahim Eden ,  Tilman Wekel ,  David Wehr ,  Ruchi Bhargava ,  David Nister

IPC: G06K9/00 ,  G01S17/931 ,  B60W60/00 ,  G01S17/89 ,  G05D1/00 ,  G06N3/04 ,  G06T19/00

Abstract: A deep neural network(s) (DNN) may be used to detect objects from sensor data of a three dimensional (3D) environment. For example, a multi-view perception DNN may include multiple constituent DNNs or stages chained together that sequentially process different views of the 3D environment. An example DNN may include a first stage that performs class segmentation in a first view (e.g., perspective view) and a second stage that performs class segmentation and/or regresses instance geometry in a second view (e.g., top-down). The DNN outputs may be processed to generate 2D and/or 3D bounding boxes and class labels for detected objects in the 3D environment. As such, the techniques described herein may be used to detect and classify animate objects and/or parts of an environment, and these detections and classifications may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

公开(公告)号：US20250045952A1

公开(公告)日：2025-02-06

申请号：US18363265

申请日：2023-08-01

Applicant: NVIDIA Corporation

Inventor： Alexander Popov ,  Nikolai Smolyanskiy ,  Ruchita Bhargava ,  Ibrahim Eden ,  Amala Sanjay Deshmukh ,  Ryan Oldja ,  Ke Chen ,  Sai Krishnan Chandrasekar ,  Minwoo Park

IPC: G06T7/73

Abstract: In various examples, systems and methods are disclosed relating to real-time multiview map generation using neural networks. A system can receive sensors images of an environment, such as images from one or more camera, RADAR, LIDAR, and/or ultrasound sensors. The system can process the sensor images using one or more neural networks, such as neural networks implementing attention structures, to detect features in the environment such as lane lines, lane dividers, wait lines, or boundaries. The system can represent the features in various views, including top-down/bird's eye view representations. The system can provide the representations for operations including map generation, map updating, perception, and object detection.

公开(公告)号：US12051206B2

公开(公告)日：2024-07-30

申请号：US16938706

申请日：2020-07-24

Applicant: NVIDIA Corporation

Inventor： Ke Chen ,  Nikolai Smolyanskiy ,  Alexey Kamenev ,  Ryan Oldja ,  Tilman Wekel ,  David Nister ,  Joachim Pehserl ,  Ibrahim Eden ,  Sangmin Oh ,  Ruchi Bhargava

IPC: G06T7/00 ,  G05D1/00 ,  G06F18/00 ,  G06F18/22 ,  G06F18/23 ,  G06T5/50 ,  G06T7/10 ,  G06T7/11 ,  G06V10/82 ,  G06V20/56 ,  G06V20/58 ,  G06V10/44

CPC classification number: G06T7/11 ,  G05D1/0088 ,  G06F18/22 ,  G06F18/23 ,  G06T5/50 ,  G06T7/10 ,  G06V10/82 ,  G06V20/56 ,  G06V20/58 ,  G06T2207/10028 ,  G06T2207/20084 ,  G06T2207/30252 ,  G06V10/454

Abstract: A deep neural network(s) (DNN) may be used to perform panoptic segmentation by performing pixel-level class and instance segmentation of a scene using a single pass of the DNN. Generally, one or more images and/or other sensor data may be stitched together, stacked, and/or combined, and fed into a DNN that includes a common trunk and several heads that predict different outputs. The DNN may include a class confidence head that predicts a confidence map representing pixels that belong to particular classes, an instance regression head that predicts object instance data for detected objects, an instance clustering head that predicts a confidence map of pixels that belong to particular instances, and/or a depth head that predicts range values. These outputs may be decoded to identify bounding shapes, class labels, instance labels, and/or range values for detected objects, and used to enable safe path planning and control of an autonomous vehicle.

公开(公告)号：US20220415059A1

公开(公告)日：2022-12-29

申请号：US17895940

申请日：2022-08-25

Applicant: NVIDIA Corporation

Inventor： Nikolai Smolyanskiy ,  Ryan Oldja ,  Ke Chen ,  Alexander Popov ,  Joachim Pehserl ,  Ibrahim Eden ,  Tilman Wekel ,  David Wehr ,  Ruchi Bhargava ,  David Nister

IPC: G06V20/58 ,  G01S17/931 ,  B60W60/00 ,  G01S17/89 ,  G05D1/00 ,  G06N3/04 ,  G06T19/00

Abstract: A deep neural network(s) (DNN) may be used to detect objects from sensor data of a three dimensional (3D) environment. For example, a multi-view perception DNN may include multiple constituent DNNs or stages chained together that sequentially process different views of the 3D environment. An example DNN may include a first stage that performs class segmentation in a first view (e.g., perspective view) and a second stage that performs class segmentation and/or regresses instance geometry in a second view (e.g., top-down). The DNN outputs may be processed to generate 2D and/or 3D bounding boxes and class labels for detected objects in the 3D environment. As such, the techniques described herein may be used to detect and classify animate objects and/or parts of an environment, and these detections and classifications may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

公开(公告)号：US20210342609A1

公开(公告)日：2021-11-04

申请号：US17377064

申请日：2021-07-15

Applicant: NVIDIA Corporation

Inventor： Nikolai Smolyanskiy ,  Ryan Oldja ,  Ke Chen ,  Alexander Popov ,  Joachim Pehserl ,  Ibrahim Eden ,  Tilman Wekel ,  David Wehr ,  Ruchi Bhargava ,  David Nister

IPC: G06K9/00 ,  G01S17/931 ,  B60W60/00 ,  G01S17/89 ,  G05D1/00 ,  G06N3/04 ,  G06T19/00

Abstract: A deep neural network(s) (DNN) may be used to detect objects from sensor data of a three dimensional (3D) environment. For example, a multi-view perception DNN may include multiple constituent DNNs or stages chained together that sequentially process different views of the 3D environment. An example DNN may include a first stage that performs class segmentation in a first view (e.g., perspective view) and a second stage that performs class segmentation and/or regresses instance geometry in a second view (e.g., top-down). The DNN outputs may be processed to generate 2D and/or 3D bounding boxes and class labels for detected objects in the 3D environment. As such, the techniques described herein may be used to detect and classify animate objects and/or parts of an environment, and these detections and classifications may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.