公开(公告)号：US11256961B2

公开(公告)日：2022-02-22

申请号：US16921012

申请日：2020-07-06

Applicant: NVIDIA Corporation

Inventor： Wei-Chih Tu ,  Ming-Yu Liu ,  Varun Jampani ,  Deqing Sun ,  Ming-Hsuan Yang ,  Jan Kautz

IPC: G06K9/00 ,  G06K9/62 ,  G06N3/04 ,  G06N3/08 ,  G06T7/11

Abstract: Segmentation is the identification of separate objects within an image. An example is identification of a pedestrian passing in front of a car, where the pedestrian is a first object and the car is a second object. Superpixel segmentation is the identification of regions of pixels within an object that have similar properties. An example is identification of pixel regions having a similar color, such as different articles of clothing worn by the pedestrian and different components of the car. A pixel affinity neural network (PAN) model is trained to generate pixel affinity maps for superpixel segmentation. The pixel affinity map defines the similarity of two points in space. In an embodiment, the pixel affinity map indicates a horizontal affinity and vertical affinity for each pixel in the image. The pixel affinity map is processed to identify the superpixels.

公开(公告)号：US11238650B2

公开(公告)日：2022-02-01

申请号：US16849962

申请日：2020-04-15

Applicant: NVIDIA Corporation

Inventor： Xueting Li ,  Sifei Liu ,  Kihwan Kim ,  Shalini De Mello ,  Varun Jampani ,  Jan Kautz

IPC: G06T15/00 ,  G06T17/20 ,  G06T7/73 ,  G06T7/40 ,  G06K9/62

Abstract: Apparatuses, systems, and techniques to identify a shape or camera pose of a three-dimensional object from a two-dimensional image of the object. In at least one embodiment, objects are identified in an image using one or more neural networks that have been trained on objects of a similar category and a three-dimensional mesh template.

公开(公告)号：US20210287430A1

公开(公告)日：2021-09-16

申请号：US16849962

申请日：2020-04-15

Applicant: NVIDIA Corporation

Inventor： Xueting Li ,  Sifei Liu ,  Kihwan Kim ,  Shalini De Mello ,  Varun Jampani ,  Jan Kautz

IPC: G06T17/20 ,  G06T7/73 ,  G06T7/40 ,  G06K9/62

Abstract: Apparatuses, systems, and techniques to identify a shape or camera pose of a three-dimensional object from a two-dimensional image of the object. In at least one embodiment, objects are identified in an image using one or more neural networks that have been trained on objects of a similar category and a three-dimensional mesh template.

公开(公告)号：US20200320401A1

公开(公告)日：2020-10-08

申请号：US16378464

申请日：2019-04-08

Applicant: NVIDIA Corporation

Inventor： Varun Jampani ,  Wei-Chih Hung ,  Sifei Liu ,  Pavlo Molchanov ,  Jan Kautz

IPC: G06N3/08 ,  G06N3/04 ,  G06T7/143 ,  G06K9/62 ,  G06F17/15

Abstract: Systems and methods to detect one or more segments of one or more objects within one or more images based, at least in part, on a neural network trained in an unsupervised manner to infer the one or more segments. Systems and methods to help train one or more neural networks to detect one or more segments of one or more objects within one or more images in an unsupervised manner.

公开(公告)号：US10776688B2

公开(公告)日：2020-09-15

申请号：US16169851

申请日：2018-10-24

Applicant: NVIDIA Corporation

Inventor： Huaizu Jiang ,  Deqing Sun ,  Varun Jampani

IPC: G06N3/04 ,  G06N3/08 ,  H04N7/01 ,  G06T7/246

Abstract: Video interpolation is used to predict one or more intermediate frames at timesteps defined between two consecutive frames. A first neural network model approximates optical flow data defining motion between the two consecutive frames. A second neural network model refines the optical flow data and predicts visibility maps for each timestep. The two consecutive frames are warped according to the refined optical flow data for each timestep to produce pairs of warped frames for each timestep. The second neural network model then fuses the pair of warped frames based on the visibility maps to produce the intermediate frame for each timestep. Artifacts caused by motion boundaries and occlusions are reduced in the predicted intermediate frames.

公开(公告)号：US20190156154A1

公开(公告)日：2019-05-23

申请号：US16188641

申请日：2018-11-13

Applicant: NVIDIA Corporation

Inventor： Wei-Chih Tu ,  Ming-Yu Liu ,  Varun Jampani ,  Deqing Sun ,  Ming-Hsuan Yang ,  Jan Kautz

IPC: G06K9/62 ,  G06N3/08 ,  G06N3/04

Abstract: Segmentation is the identification of separate objects within an image. An example is identification of a pedestrian passing in front of a car, where the pedestrian is a first object and the car is a second object. Superpixel segmentation is the identification of regions of pixels within an object that have similar properties An example is identification of pixel regions having a similar color, such as different articles of clothing worn by the pedestrian and different components of the car. A pixel affinity neural network (PAN) model is trained to generate pixel affinity maps for superpixel segmentation. The pixel affinity map defines the similarity of two points in space. In an embodiment, the pixel affinity map indicates a horizonal affinity and vertical affinity for each pixel in the image. The pixel affinity map is processed to identify the superpixels.

公开(公告)号：US20190138889A1

公开(公告)日：2019-05-09

申请号：US16169851

申请日：2018-10-24

Applicant: NVIDIA Corporation

Inventor： Huaizu Jiang ,  Deqing Sun ,  Varun Jampani

IPC: G06N3/04 ,  G06N3/08 ,  G06T7/246 ,  H04N7/01

Abstract: Video interpolation is used to predict one or more intermediate frames at timesteps defined between two consecutive frames. A first neural network model approximates optical flow data defining motion between the two consecutive frames. A second neural network model refines the optical flow data and predicts visibility maps for each timestep. The two consecutive frames are warped according to the refined optical flow data for each timestep to produce pairs of warped frames for each timestep. The second neural network model then fuses the pair of warped frames based on the visibility maps to produce the intermediate frame for each timestep. Artifacts caused by motion boundaries and occlusions are reduced in the predicted intermediate frames.

公开(公告)号：US11748887B2

公开(公告)日：2023-09-05

申请号：US16378464

申请日：2019-04-08

Applicant: NVIDIA Corporation

Inventor： Varun Jampani ,  Wei-Chih Hung ,  Sifei Liu ,  Pavlo Molchanov ,  Jan Kautz

IPC: G06V10/00 ,  G06T7/11 ,  G06T7/143 ,  G06F17/15 ,  G06N3/088 ,  G06F18/40 ,  G06N3/045 ,  G06N3/047 ,  G06V10/764 ,  G06V10/82 ,  G06V10/94 ,  G06V20/40

CPC classification number: G06T7/11 ,  G06F17/15 ,  G06F18/40 ,  G06N3/045 ,  G06N3/047 ,  G06N3/088 ,  G06T7/143 ,  G06V10/764 ,  G06V10/82 ,  G06V10/945 ,  G06V20/41

Abstract: Systems and methods to detect one or more segments of one or more objects within one or more images based, at least in part, on a neural network trained in an unsupervised manner to infer the one or more segments. Systems and methods to help train one or more neural networks to detect one or more segments of one or more objects within one or more images in an unsupervised manner.

公开(公告)号：US11676284B2

公开(公告)日：2023-06-13

申请号：US16825192

申请日：2020-03-20

Applicant: Nvidia Corporation

Inventor： David Jesus Acuna Marrero ,  Towaki Takikawa ,  Varun Jampani ,  Sanja Fidler

IPC: G06K9/00 ,  G06T7/12 ,  G06V20/56 ,  G06F18/25 ,  G06V10/764 ,  G06V10/80 ,  G06V10/82 ,  G06V10/44 ,  G06V10/20

CPC classification number: G06T7/12 ,  G06F18/253 ,  G06V10/255 ,  G06V10/454 ,  G06V10/764 ,  G06V10/806 ,  G06V10/82 ,  G06V20/56 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/30252

Abstract: Various types of image analysis benefit from a multi-stream architecture that allows the analysis to consider shape data. A shape stream can process image data in parallel with a primary stream, where data from layers of a network in the primary stream is provided as input to a network of the shape stream. The shape data can be fused with the primary analysis data to produce more accurate output, such as to produce accurate boundary information when the shape data is used with semantic segmentation data produced by the primary stream. A gate structure can be used to connect the intermediate layers of the primary and shape streams, using higher level activations to gate lower level activations in the shape stream. Such a gate structure can help focus the shape stream on the relevant information and reduces any additional weight of the shape stream.

公开(公告)号：US11182649B2

公开(公告)日：2021-11-23

申请号：US17119971

申请日：2020-12-11

Applicant: NVIDIA Corporation

Inventor： Jonathan Tremblay ,  Aayush Prakash ,  Mark A. Brophy ,  Varun Jampani ,  Cem Anil ,  Stanley Thomas Birchfield ,  Thang Hong To ,  David Jesus Acuna Marrero

IPC: G06K9/62 ,  G06T15/04 ,  G06T15/50 ,  G06T15/20 ,  G06N3/04 ,  G06N3/08

Abstract: Training deep neural networks requires a large amount of labeled training data. Conventionally, labeled training data is generated by gathering real images that are manually labelled which is very time-consuming. Instead of manually labelling a training dataset, domain randomization technique is used generate training data that is automatically labeled. The generated training data may be used to train neural networks for object detection and segmentation (labelling) tasks. In an embodiment, the generated training data includes synthetic input images generated by rendering three-dimensional (3D) objects of interest in a 3D scene. In an embodiment, the generated training data includes synthetic input images generated by rendering 3D objects of interest on a 2D background image. The 3D objects of interest are objects that a neural network is trained to detect and/or label.