公开(公告)号：US11604967B2

公开(公告)日：2023-03-14

申请号：US17356140

申请日：2021-06-23

Applicant: NVIDIA Corporation

Inventor： Nikolai Smolyanskiy ,  Alexey Kamenev ,  Stan Birchfield

IPC: G01S17/88 ,  G01S17/894 ,  G06N3/02 ,  G06N3/084 ,  G06T7/50 ,  G06T7/80 ,  G06N3/04 ,  G06T7/593 ,  G06N3/088 ,  G06T1/20 ,  G06K9/62 ,  G06N3/063 ,  G01S17/86 ,  G01S17/89

Abstract: Various examples of the present disclosure include a stereoscopic deep neural network (DNN) that produces accurate and reliable results in real-time. Both LIDAR data (supervised training) and photometric error (unsupervised training) may be used to train the DNN in a semi-supervised manner. The stereoscopic DNN may use an exponential linear unit (ELU) activation function to increase processing speeds, as well as a machine learned argmax function that may include a plurality of convolutional layers having trainable parameters to account for context. The stereoscopic DNN may further include layers having an encoder/decoder architecture, where the encoder portion of the layers may include a combination of three-dimensional convolutional layers followed by two-dimensional convolutional layers.

公开(公告)号：US20210026355A1

公开(公告)日：2021-01-28

申请号：US16938706

申请日：2020-07-24

Applicant: NVIDIA Corporation

Inventor： Ke Chen ,  Nikolai Smolyanskiy ,  Alexey Kamenev ,  Ryan Oldja ,  Tilman Wekel ,  David Nister ,  Joachim Pehserl ,  Ibrahim Eden ,  Sangmin Oh ,  Ruchi Bhargava

IPC: G05D1/00 ,  G06T7/10 ,  G06T5/50 ,  G06K9/00 ,  G06K9/62

Abstract: A deep neural network(s) (DNN) may be used to perform panoptic segmentation by performing pixel-level class and instance segmentation of a scene using a single pass of the DNN. Generally, one or more images and/or other sensor data may be stitched together, stacked, and/or combined, and fed into a DNN that includes a common trunk and several heads that predict different outputs. The DNN may include a class confidence head that predicts a confidence map representing pixels that belong to particular classes, an instance regression head that predicts object instance data for detected objects, an instance clustering head that predicts a confidence map of pixels that belong to particular instances, and/or a depth head that predicts range values. These outputs may be decoded to identify bounding shapes, class labels, instance labels, and/or range values for detected objects, and used to enable safe path planning and control of an autonomous vehicle.

公开(公告)号：US20190295282A1

公开(公告)日：2019-09-26

申请号：US16356439

申请日：2019-03-18

Applicant: NVIDIA Corporation

Inventor： Nikolai Smolyanskiy ,  Alexey Kamenev ,  Stan Birchfield

IPC: G06T7/593 ,  G06N3/08 ,  G06N3/04 ,  G06K9/62 ,  G06T1/20

Abstract: Various examples of the present disclosure include a stereoscopic deep neural network (DNN) that produces accurate and reliable results in real-time. Both LIDAR data (supervised training) and photometric error (unsupervised training) may be used to train the DNN in a semi-supervised manner. The stereoscopic DNN may use an exponential linear unit (ELU) activation function to increase processing speeds, as well as a machine learned argmax function that may include a plurality of convolutional layers having trainable parameters to account for context. The stereoscopic DNN may further include layers having an encoder/decoder architecture, where the encoder portion of the layers may include a combination of three-dimensional convolutional layers followed by two-dimensional convolutional layers.

公开(公告)号：US12039436B2

公开(公告)日：2024-07-16

申请号：US18160694

申请日：2023-01-27

Applicant: NVIDIA Corporation

Inventor： Nikolai Smolyanskiy ,  Alexey Kamenev ,  Stan Birchfield

IPC: G06T7/50 ,  G01S17/86 ,  G01S17/89 ,  G06F18/22 ,  G06N3/02 ,  G06N3/045 ,  G06N3/048 ,  G06N3/063 ,  G06N3/084 ,  G06N3/088 ,  G06T1/20 ,  G06T7/593

CPC classification number: G06N3/063 ,  G01S17/86 ,  G01S17/89 ,  G06F18/22 ,  G06N3/045 ,  G06N3/048 ,  G06N3/084 ,  G06N3/088 ,  G06T1/20 ,  G06T7/593 ,  G06T2207/10012 ,  G06T2207/10052 ,  G06T2207/20084

Abstract: Various examples of the present disclosure include a stereoscopic deep neural network (DNN) that produces accurate and reliable results in real-time. Both LIDAR data (supervised training) and photometric error (unsupervised training) may be used to train the DNN in a semi-supervised manner. The stereoscopic DNN may use an exponential linear unit (ELU) activation function to increase processing speeds, as well as a machine learned argmax function that may include a plurality of convolutional layers having trainable parameters to account for context. The stereoscopic DNN may further include layers having an encoder/decoder architecture, where the encoder portion of the layers may include a combination of three-dimensional convolutional layers followed by two-dimensional convolutional layers.

公开(公告)号：US12001958B2

公开(公告)日：2024-06-04

申请号：US16824199

申请日：2020-03-19

Applicant: NVIDIA Corporation

Inventor： Alexey Kamenev ,  Nikolai Smolyanskiy ,  Ishwar Kulkarni ,  Ollin Boer Bohan ,  Fangkai Yang ,  Alperen Degirmenci ,  Ruchi Bhargava ,  Urs Muller ,  David Nister ,  Rotem Aviv

IPC: G06N3/088 ,  G06N3/044 ,  G06N3/045

CPC classification number: G06N3/088 ,  G06N3/044 ,  G06N3/045

Abstract: In various examples, past location information corresponding to actors in an environment and map information may be applied to a deep neural network (DNN)—such as a recurrent neural network (RNN)—trained to compute information corresponding to future trajectories of the actors. The output of the DNN may include, for each future time slice the DNN is trained to predict, a confidence map representing a confidence for each pixel that an actor is present and a vector field representing locations of actors in confidence maps for prior time slices. The vector fields may thus be used to track an object through confidence maps for each future time slice to generate a predicted future trajectory for each actor. The predicted future trajectories, in addition to tracked past trajectories, may be used to generate full trajectories for the actors that may aid an ego-vehicle in navigating the environment.

公开(公告)号：US20240059285A1

公开(公告)日：2024-02-22

申请号：US17891587

申请日：2022-08-19

Applicant: NVIDIA Corporation

Inventor： Julia Ng ,  Jian Wei Leong ,  Nikolai Smolyanskiy ,  Yizhou Wang ,  Fangkai Yang ,  Nianfeng Wan ,  Chang Liu

IPC: B60W30/14 ,  B60W60/00 ,  B60W50/00

CPC classification number: B60W30/14 ,  B60W60/001 ,  B60W50/0097 ,  B60W2520/00 ,  B60W2556/00

Abstract: In various examples, techniques for using future trajectory predictions for adaptive cruise control (ACC) are described. For instance, a vehicle may determine a future path(s) of the vehicle and a future path(s) of an object(s). The vehicle may then use a speed profile(s) and the future path(s) to determine a trajectory(ies) for the vehicle. The vehicle may then select a trajectory, such as based on the future path(s) of the object(s). Based on the trajectory, ACC of the vehicle may cause the vehicle to navigate at a speed or a velocity. This way, the vehicle is able to continue using ACC even when the driver makes a maneuver(s) or the system determined to make a maneuver, such as switching lanes or choosing a lane when a road splits.

公开(公告)号：US20230281847A1

公开(公告)日：2023-09-07

申请号：US17592096

申请日：2022-02-03

Applicant: NVIDIA Corporation

Inventor： Yiran Zhong ,  Charles Loop ,  Nikolai Smolyanskiy ,  Ke Chen ,  Stan Birchfield ,  Alexander Popov

IPC: G06T7/55 ,  G06T7/70 ,  G06V10/46

CPC classification number: G06T7/55 ,  G06T7/70 ,  G06V10/462 ,  G06T2207/20081 ,  G06T2207/30252

Abstract: In various examples, methods and systems are provided for estimating depth values for images (e.g., from a monocular sequence). Disclosed approaches may define a search space of potential pixel matches between two images using one or more depth hypothesis planes based at least on a camera pose associated with one or more cameras used to generate the images. A machine learning model(s) may use this search space to predict likelihoods of correspondence between one or more pixels in the images. The predicted likelihoods may be used to compute depth values for one or more of the images. The predicted depth values may be transmitted and used by a machine to perform one or more operations.

公开(公告)号：US20230260136A1

公开(公告)日：2023-08-17

申请号：US17672402

申请日：2022-02-15

Applicant: NVIDIA Corporation

Inventor： Jens Christian Bo Joergensen ,  Ollin Boer Bohan ,  Joachim Pehserl ,  Nikolai Smolyanskiy

IPC: G06T7/254 ,  G06V10/44

CPC classification number: G06T7/254 ,  G06V10/454 ,  G06T2207/10028 ,  G06T2207/20084 ,  G06V2201/07

Abstract: In various examples, systems and methods of the present disclosure detect and/or track objects in an environment using projection images generated from LiDAR. For example, a machine learning model—such as a deep neural network (DNN)—may be used to compute a motion mask indicative of motion corresponding to points representing objects in an environment. Various input channels may be provided as input to the machine learning model to compute a motion mask. One or more comparison images may be generated based on comparing depth values projected from a current range image to a coordinate space of a previous range image to depth values of the previous range image. The machine learning model may use the one or more projection images, the one or more comparison images, and/or the one or more range images to compute a motion mask and/or a motion vector output representation.

公开(公告)号：US11532168B2

公开(公告)日：2022-12-20

申请号：US16915346

申请日：2020-06-29

Applicant: NVIDIA Corporation

Inventor： Nikolai Smolyanskiy ,  Ryan Oldja ,  Ke Chen ,  Alexander Popov ,  Joachim Pehserl ,  Ibrahim Eden ,  Tilman Wekel ,  David Wehr ,  Ruchi Bhargava ,  David Nister

IPC: G06V20/58 ,  G01S17/931 ,  B60W60/00 ,  G01S17/89 ,  G05D1/00 ,  G06N3/04 ,  G06T19/00

Abstract: A deep neural network(s) (DNN) may be used to detect objects from sensor data of a three dimensional (3D) environment. For example, a multi-view perception DNN may include multiple constituent DNNs or stages chained together that sequentially process different views of the 3D environment. An example DNN may include a first stage that performs class segmentation in a first view (e.g., perspective view) and a second stage that performs class segmentation and/or regresses instance geometry in a second view (e.g., top-down). The DNN outputs may be processed to generate 2D and/or 3D bounding boxes and class labels for detected objects in the 3D environment. As such, the techniques described herein may be used to detect and classify animate objects and/or parts of an environment, and these detections and classifications may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

公开(公告)号：US20220138568A1

公开(公告)日：2022-05-05

申请号：US17453055

申请日：2021-11-01

Applicant: NVIDIA Corporation

Inventor： Nikolai Smolyanskiy ,  Alexey Kamenev ,  Lirui Wang ,  David Nister ,  Ollin Boer Bohan ,  Ishwar Kulkarni ,  Fangkai Yang ,  Julia Ng ,  Alperen Degirmenci ,  Ruchi Bhargava ,  Rotem Aviv

IPC: G06N3/08 ,  G06N3/04

Abstract: In various examples, reinforcement learning is used to train at least one machine learning model (MLM) to control a vehicle by leveraging a deep neural network (DNN) trained on real-world data by using imitation learning to predict movements of one or more actors to define a world model. The DNN may be trained from real-world data to predict attributes of actors, such as locations and/or movements, from input attributes. The predictions may define states of the environment in a simulator, and one or more attributes of one or more actors input into the DNN may be modified or controlled by the simulator to simulate conditions that may otherwise be unfeasible. The MLM(s) may leverage predictions made by the DNN to predict one or more actions for the vehicle.