公开(公告)号：US20210097346A1

公开(公告)日：2021-04-01

申请号：US17119971

申请日：2020-12-11

Applicant: NVIDIA Corporation

Inventor： Jonathan Tremblay ,  Aayush Prakash ,  Mark A. Brophy ,  Varun Jampani ,  Cem Anil ,  Stanley Thomas Birchfield ,  Thang Hong To ,  David Jesus Acuna Marrero

IPC: G06K9/62 ,  G06T15/04 ,  G06T15/50 ,  G06T15/20

Abstract: Training deep neural networks requires a large amount of labeled training data. Conventionally, labeled training data is generated by gathering real images that are manually labelled which is very time-consuming. Instead of manually labelling a training dataset, domain randomization technique is used generate training data that is automatically labeled. The generated training data may be used to train neural networks for object detection and segmentation (labelling) tasks. In an embodiment, the generated training data includes synthetic input images generated by rendering three-dimensional (3D) objects of interest in a 3D scene. In an embodiment, the generated training data includes synthetic input images generated by rendering 3D objects of interest on a 2D background image. The 3D objects of interest are objects that a neural network is trained to detect and/or label.

公开(公告)号：US20250139783A1

公开(公告)日：2025-05-01

申请号：US18943472

申请日：2024-11-11

Applicant: Nvidia Corporation

Inventor： David Jesus Acuna Marrero ,  Towaki Takikawa ,  Varun Jampani ,  Sanja Fidler

IPC: G06T7/12 ,  G06F18/25 ,  G06V10/20 ,  G06V10/44 ,  G06V10/764 ,  G06V10/80 ,  G06V10/82 ,  G06V20/56

Abstract: Various types of image analysis benefit from a multi-stream architecture that allows the analysis to consider shape data. A shape stream can process image data in parallel with a primary stream, where data from layers of a network in the primary stream is provided as input to a network of the shape stream. The shape data can be fused with the primary analysis data to produce more accurate output, such as to produce accurate boundary information when the shape data is used with semantic segmentation data produced by the primary stream. A gate structure can be used to connect the intermediate layers of the primary and shape streams, using higher level activations to gate lower level activations in the shape stream. Such a gate structure can help focus the shape stream on the relevant information and reduces any additional weight of the shape stream.

公开(公告)号：US11328169B2

公开(公告)日：2022-05-10

申请号：US16353835

申请日：2019-03-14

Applicant: NVIDIA Corporation

Inventor： Sifei Liu ,  Shalini De Mello ,  Jinwei Gu ,  Varun Jampani ,  Jan Kautz

IPC: G06K9/62 ,  G06V20/40 ,  G06T7/90 ,  G06T5/00 ,  G06T7/10 ,  G06N3/08 ,  G06N3/04 ,  G06T5/50

Abstract: A temporal propagation network (TPN) system learns the affinity matrix for video image processing tasks. An affinity matrix is a generic matrix that defines the similarity of two points in space. The TPN system includes a guidance neural network model and a temporal propagation module and is trained for a particular computer vision task to propagate visual properties from a key-frame represented by dense data (color), to another frame that is represented by coarse data (grey-scale). The guidance neural network model generates an affinity matrix referred to as a global transformation matrix from task-specific data for the key-frame and the other frame. The temporal propagation module applies the global transformation matrix to the key-frame property data to produce propagated property data (color) for the other frame. For example, the TPN system may be used to colorize several frames of greyscale video using a single manually colorized key-frame.

公开(公告)号：US20210073575A1

公开(公告)日：2021-03-11

申请号：US17081805

申请日：2020-10-27

Applicant: NVIDIA Corporation

Inventor： Sifei Liu ,  Shalini De Mello ,  Jinwei Gu ,  Varun Jampani ,  Jan Kautz

IPC: G06K9/62 ,  G06K9/00 ,  G06T7/90 ,  G06T5/00 ,  G06T7/10 ,  G06N3/08 ,  G06N3/04 ,  G06T5/50

Abstract: A temporal propagation network (TPN) system learns the affinity matrix for video image processing tasks. An affinity matrix is a generic matrix that defines the similarity of two points in space. The TPN system includes a guidance neural network model and a temporal propagation module and is trained for a particular computer vision task to propagate visual properties from a key-frame represented by dense data (color), to another frame that is represented by coarse data (grey-scale). The guidance neural network model generates an affinity matrix referred to as a global transformation matrix from task-specific data for the key-frame and the other frame. The temporal propagation module applies the global transformation matrix to the key-frame property data to produce propagated property data (color) for the other frame. For example, the TPN system may be used to colorize several frames of greyscale video using a single manually colorized key-frame.

公开(公告)号：US20200084427A1

公开(公告)日：2020-03-12

申请号：US16569104

申请日：2019-09-12

Applicant: NVIDIA Corporation

Inventor： Deqing Sun ,  Varun Jampani ,  Erik Gundersen Learned-Miller ,  Huaizu Jiang

IPC: H04N13/122 ,  H04N13/128 ,  G06N3/08

Abstract: Scene flow represents the three-dimensional (3D) structure and movement of objects in a video sequence in three dimensions from frame-to-frame and is used to track objects and estimate speeds for autonomous driving applications. Scene flow is recovered by a neural network system from a video sequence captured from at least two viewpoints (e.g., cameras), such as a left-eye and right-eye of a viewer. An encoder portion of the system extracts features from frames of the video sequence. The features are input to a first decoder to predict optical flow and a second decoder to predict disparity. The optical flow represents pixel movement in (x,y) and the disparity represents pixel movement in z (depth). When combined, the optical flow and disparity represent the scene flow.

公开(公告)号：US20190251397A1

公开(公告)日：2019-08-15

申请号：US16256820

申请日：2019-01-24

Applicant: NVIDIA Corporation

Inventor： Jonathan Tremblay ,  Aayush Prakash ,  Mark A. Brophy ,  Varun Jampani ,  Cem Anil ,  Stanley Thomas Birchfield ,  Thang Hong To ,  David Jesus Acuna Marrero

IPC: G06K9/62 ,  G06T15/04 ,  G06T15/50 ,  G06T15/20

CPC classification number: G06K9/6257 ,  G06K9/6228 ,  G06K2209/21 ,  G06N3/04 ,  G06N3/084 ,  G06T15/04 ,  G06T15/20 ,  G06T15/50 ,  G06T2210/12

Abstract: Training deep neural networks requires a large amount of labeled training data. Conventionally, labeled training data is generated by gathering real images that are manually labelled which is very time-consuming. Instead of manually labelling a training dataset, domain randomization technique is used generate training data that is automatically labeled. The generated training data may be used to train neural networks for object detection and segmentation (labelling) tasks. In an embodiment, the generated training data includes synthetic input images generated by rendering three-dimensional (3D) objects of interest in a 3D scene. In an embodiment, the generated training data includes synthetic input images generated by rendering 3D objects of interest on a 2D background image. The 3D objects of interest are objects that a neural network is trained to detect and/or label.

公开(公告)号：US20190213439A1

公开(公告)日：2019-07-11

申请号：US16353835

申请日：2019-03-14

Applicant: NVIDIA Corporation

Inventor： Sifei Liu ,  Shalini De Mello ,  Jinwei Gu ,  Varun Jampani ,  Jan Kautz

IPC: G06K9/62 ,  G06K9/00 ,  G06T7/90 ,  G06T5/00 ,  G06T7/10 ,  G06T5/50 ,  G06N3/08 ,  G06N3/04

CPC classification number: G06K9/6215 ,  G06K9/00744 ,  G06K9/6256 ,  G06N3/04 ,  G06N3/08 ,  G06N3/084 ,  G06T5/009 ,  G06T5/50 ,  G06T7/10 ,  G06T7/90 ,  G06T2207/10016 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/20208

Abstract: A temporal propagation network (TPN) system learns the affinity matrix for video image processing tasks. An affinity matrix is a generic matrix that defines the similarity of two points in space. The TPN system includes a guidance neural network model and a temporal propagation module and is trained for a particular computer vision task to propagate visual properties from a key-frame represented by dense data (color), to another frame that is represented by coarse data (grey-scale). The guidance neural network model generates an affinity matrix referred to as a global transformation matrix from task-specific data for the key-frame and the other frame. The temporal propagation module applies the global transformation matrix to the key-frame property data to produce propagated property data (color) for the other frame. For example, the TPN system may be used to colorize several frames of greyscale video using a single manually colorized key-frame.

公开(公告)号：US11907846B2

公开(公告)日：2024-02-20

申请号：US17017597

申请日：2020-09-10

Applicant: NVIDIA CORPORATION

Inventor： Sifei Liu ,  Shalini De Mello ,  Varun Jampani ,  Jan Kautz ,  Xueting Li

IPC: G06K9/36 ,  G06N3/084 ,  G06F18/22 ,  G06F18/20 ,  G06F18/214 ,  G06F18/21 ,  G06N3/045 ,  G06T17/00 ,  G06V10/82

CPC classification number: G06N3/084 ,  G06F18/214 ,  G06F18/2163 ,  G06F18/22 ,  G06F18/29 ,  G06N3/045 ,  G06T17/00 ,  G06V10/82

Abstract: One embodiment of the present invention sets forth a technique for performing spatial propagation. The technique includes generating a first directed acyclic graph (DAG) by connecting spatially adjacent points included in a set of unstructured points via directed edges along a first direction. The technique also includes applying a first set of neural network layers to one or more images associated with the set of unstructured points to generate (i) a set of features for the set of unstructured points and (ii) a set of pairwise affinities between the spatially adjacent points connected by the directed edges. The technique further includes generating a set of labels for the set of unstructured points by propagating the set of features across the first DAG based on the set of pairwise affinities.

公开(公告)号：US20220254029A1

公开(公告)日：2022-08-11

申请号：US17500338

申请日：2021-10-13

Applicant: NVIDIA Corporation

Inventor： Eugene Vorontsov ,  Wonmin Byeon ,  Shalini De Mello ,  Varun Jampani ,  Ming-Yu Liu ,  Pavlo Molchanov

IPC: G06T7/11 ,  G06K9/62

Abstract: The neural network includes an encoder, a common decoder, and a residual decoder. The encoder encodes input images into a latent space. The latent space disentangles unique features from other common features. The common decoder decodes common features resident in the latent space to generate translated images which lack the unique features. The residual decoder decodes unique features resident in the latent space to generate image deltas corresponding to the unique features. The neural network combines the translated images with the image deltas to generate combined images that may include both common features and unique features. The combined images can be used to drive autoencoding. Once training is complete, the residual decoder can be modified to generate segmentation masks that indicate any regions of a given input image where a unique feature resides.

公开(公告)号：US20220139037A1

公开(公告)日：2022-05-05

申请号：US17578051

申请日：2022-01-18

Applicant: NVIDIA Corporation

Inventor： Xueting Li ,  Sifei Liu ,  Kihwan Kim ,  Shalini De Mello ,  Varun Jampani ,  Jan Kautz

IPC: G06T17/20 ,  G06T7/73 ,  G06T7/40 ,  G06K9/62

Abstract: Apparatuses, systems, and techniques to identify a shape or camera pose of a three-dimensional object from a two-dimensional image of the object. In at least one embodiment, objects are identified in an image using one or more neural networks that have been trained on objects of a similar category and a three-dimensional mesh template.