公开(公告)号：US20240135630A1

公开(公告)日：2024-04-25

申请号：US18485225

申请日：2023-10-11

Applicant: NVIDIA Corporation

Inventor： Koki Nagano ,  Eric Ryan Wong Chan ,  Tero Tapani Karras ,  Shalini De Mello ,  Miika Samuli Aittala ,  Matthew Aaron Wong Chan

IPC: G06T15/06 ,  G06T5/00 ,  G06T5/50 ,  G06V10/44 ,  G06V10/771

CPC classification number: G06T15/06 ,  G06T5/002 ,  G06T5/50 ,  G06V10/44 ,  G06V10/771 ,  G06T2207/20084 ,  G06T2207/20221

Abstract: A method and system for performing novel image synthesis using generative networks are provided. The encoder-based model is trained to infer a 3D representation of an input image. A feature image is then generated using volume rendering techniques in accordance with the 3D representation. The feature image is then concatenated with a noisy image and processed by a denoiser network to predict an output image from a novel viewpoint that is consistent with the input image. The denoiser network can be a modified Noise Conditional Score Network (NCSN). In some embodiments, multiple input images or keyframes can be provided as input, and a different 3D representation is generated for each input image. The feature image is then generated, during volume rendering, by sampling each of the 3D representations and applying a mean-pooling operation to generate an aggregate feature image.