公开(公告)号：US11748940B1

公开(公告)日：2023-09-05

申请号：US17498505

申请日：2021-10-11

Applicant: META PLATFORMS TECHNOLOGIES, LLC

Inventor： Wenqi Xian ,  Jia-Bin Huang ,  Johannes Peter Kopf ,  Changil Kim

IPC: G06T15/20 ,  G06N3/08

CPC classification number: G06T15/205 ,  G06N3/08

Abstract: In one embodiment, a computing system may determine a view position, a view direction, and a time with respect to a scene. The system may access a spatiotemporal representation of the scene generated based on (1) a monocular video including images each capturing at least a portion of the scene at a corresponding time and (2) depth values of the portion of the scene captured by each image. The system may generate an image based on the view position, the view direction, the time, and the spatiotemporal representation. A pixel value of the image corresponding to the view position may be determined based on volume densities and color values at sampling locations along the view direction and at the time in the spatiotemporal representation. The system may output the image to the display, representing the scene at the time as viewed from the view position and in the view direction.

公开(公告)号：US12243273B2

公开(公告)日：2025-03-04

申请号：US17571285

申请日：2022-01-07

Applicant: META PLATFORMS TECHNOLOGIES, LLC

Inventor： Zhaoyang Lv ,  Miroslava Slavcheva ,  Tianye Li ,  Michael Zollhoefer ,  Simon Gareth Green ,  Tanner Schmidt ,  Michael Goesele ,  Steven John Lovegrove ,  Christoph Lassner ,  Changil Kim

IPC: G06T7/00

Abstract: In one embodiment, a method includes initializing latent codes respectively associated with times associated with frames in a training video of a scene captured by a camera. For each of the frames, a system (1) generates rendered pixel values for a set of pixels in the frame by querying NeRF using the latent code associated with the frame, a camera viewpoint associated with the frame, and ray directions associated with the set of pixels, and (2) updates the latent code associated with the frame and the NeRF based on comparisons between the rendered pixel values and original pixel values for the set of pixels. Once trained, the system renders output frames for an output video of the scene, wherein each output frame is rendered by querying the updated NeRF using one of the updated latent codes corresponding to a desired time associated with the output frame.