公开(公告)号：US20230325968A1

公开(公告)日：2023-10-12

申请号：US17714356

申请日：2022-04-06

Applicant: Adobe Inc.

Inventor： Simon Niklaus ,  Ping Hu

IPC: G06T3/40 ,  G06T3/00 ,  G06T5/50

CPC classification number: G06T3/4007 ,  G06T3/0093 ,  G06T5/50 ,  G06T2207/20221 ,  G06T2207/20016

Abstract: Digital synthesis techniques are described to synthesize a digital image at a target time between a first digital image and a second digital image. To begin, an optical flow generation module is employed to generate optical flows. The digital images and optical flows are then received as an input by a motion refinement system. The motion refinement system is configured to generate data describing many-to-many relationships mapped for pixels in the plurality of digital images and reliability scores of the many-to-many relationships. The reliability scores are then used to resolve overlaps of pixels that are mapped to a same location by a synthesis module to generate a synthesized digital image.

公开(公告)号：US20230326028A1

公开(公告)日：2023-10-12

申请号：US17658873

申请日：2022-04-12

Applicant: Adobe Inc.

Inventor： Jianming Zhang ,  Soo Ye Kim ,  Simon Niklaus ,  Yifei Fan ,  Su Chen ,  Zhe Lin

IPC: G06T7/11 ,  G06T7/215 ,  G06T7/50

CPC classification number: G06T7/11 ,  G06T2207/20084 ,  G06T7/50 ,  G06T7/215

Abstract: The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing machine learning models to generate refined depth maps of digital images utilizing digital segmentation masks. In particular, in one or more embodiments, the disclosed systems generate a depth map for a digital image utilizing a depth estimation machine learning model, determine a digital segmentation mask for the digital image, and generate a refined depth map from the depth map and the digital segmentation mask utilizing a depth refinement machine learning model. In some embodiments, the disclosed systems generate first and second intermediate depth maps using the digital segmentation mask and an inverse digital segmentation mask and merger the first and second intermediate depth maps to generate the refined depth map.

公开(公告)号：US20220301252A1

公开(公告)日：2022-09-22

申请号：US17204571

申请日：2021-03-17

Applicant: ADOBE INC.

Inventor： Oliver Wang ,  Simon Niklaus ,  Zhengqi Li

IPC: G06T15/04 ,  G06T7/557 ,  G06T7/90 ,  G06T15/50

Abstract: Embodiments of the technology described herein, provide a view and time synthesis of dynamic scenes captured by a camera. The technology described herein represents a dynamic scene as a continuous function of both space and time. The technology may parameterize this function with a deep neural network (a multi-layer perceptron (MLP)), and perform rendering using volume tracing. At a very high level, a dynamic scene depicted in the video may be used to train the MLP. Once trained, the MLP is able to synthesize a view of the scene at a time and/or camera pose not found in the video through prediction. As used herein, a dynamic scene comprises one or more moving objects.

公开(公告)号：US20230326044A1

公开(公告)日：2023-10-12

申请号：US17714373

申请日：2022-04-06

Applicant: Adobe Inc.

Inventor： Simon Niklaus ,  Jiawen Chen

IPC: G06T7/269 ,  G06T11/00 ,  G06T3/40

CPC classification number: G06T7/269 ,  G06T11/00 ,  G06T3/40 ,  G06T2207/20221 ,  G06T2207/20084

Abstract: Digital image synthesis techniques are described that leverage splatting, i.e., forward warping. In one example, a first digital image and a first optical flow are received by a digital image synthesis system. A first splat metric and a first merge metric are constructed by the digital image synthesis system that defines a weighted map of respective pixels. From this, the digital image synthesis system produces a first warped optical flow and a first warp merge metric corresponding to an interpolation instant by forward warping the first optical flow based on the splat metric and the merge metric. A first warped digital image corresponding to the interpolation instant is formed by the digital image synthesis system by backward warping the first digital image based on the first warped optical flow.

公开(公告)号：US11443481B1

公开(公告)日：2022-09-13

申请号：US17186522

申请日：2021-02-26

Applicant: Adobe Inc.

Inventor： Wei Yin ,  Jianming Zhang ,  Oliver Wang ,  Simon Niklaus ,  Mai Long ,  Su Chen

IPC: G06T17/00 ,  G06T7/80 ,  G06N3/04 ,  G06V20/00

Abstract: This disclosure describes implementations of a three-dimensional (3D) scene recovery system that reconstructs a 3D scene representation of a scene portrayed in a single digital image. For instance, the 3D scene recovery system trains and utilizes a 3D point cloud model to recover accurate intrinsic camera parameters from a depth map of the digital image. Additionally, the 3D point cloud model may include multiple neural networks that target specific intrinsic camera parameters. For example, the 3D point cloud model may include a depth 3D point cloud neural network that recovers the depth shift as well as include a focal length 3D point cloud neural network that recovers the camera focal length. Further, the 3D scene recovery system may utilize the recovered intrinsic camera parameters to transform the single digital image into an accurate and realistic 3D scene representation, such as a 3D point cloud.

公开(公告)号：US11017586B2

公开(公告)日：2021-05-25

申请号：US16388187

申请日：2019-04-18

Applicant: ADOBE INC.

Inventor： Mai Long ,  Simon Niklaus ,  Jimei Yang

IPC: G06T15/20 ,  G06T7/10 ,  G06N3/04

Abstract: Systems and methods are described for generating a three dimensional (3D) effect from a two dimensional (2D) image. The methods may include generating a depth map based on a 2D image, identifying a camera path, generating one or more extremal views based on the 2D image and the camera path, generating a global point cloud by inpainting occlusion gaps in the one or more extremal views, generating one or more intermediate views based on the global point cloud and the camera path, and combining the one or more extremal views and the one or more intermediate views to produce a 3D motion effect.

公开(公告)号：US12039657B2

公开(公告)日：2024-07-16

申请号：US17204571

申请日：2021-03-17

Applicant: ADOBE INC.

Inventor： Oliver Wang ,  Simon Niklaus ,  Zhengqi Li

IPC: G06T15/04 ,  G06T7/557 ,  G06T7/90 ,  G06T15/50

CPC classification number: G06T15/04 ,  G06T7/557 ,  G06T7/90 ,  G06T15/506 ,  G06T2207/10016 ,  G06T2207/20084 ,  G06T2207/30244

Abstract: Embodiments of the technology described herein, provide a view and time synthesis of dynamic scenes captured by a camera. The technology described herein represents a dynamic scene as a continuous function of both space and time. The technology may parameterize this function with a deep neural network (a multi-layer perceptron (MLP)), and perform rendering using volume tracing. At a very high level, a dynamic scene depicted in the video may be used to train the MLP. Once trained, the MLP is able to synthesize a view of the scene at a time and/or camera pose not found in the video through prediction. As used herein, a dynamic scene comprises one or more moving objects.

公开(公告)号：US11871145B2

公开(公告)日：2024-01-09

申请号：US17223945

申请日：2021-04-06

Applicant: Adobe Inc.

Inventor： Simon Niklaus ,  Oliver Wang ,  Long Mai

IPC: H04N7/01 ,  G06N3/08 ,  G06T5/20 ,  G06T5/50 ,  G06T5/00 ,  G06N3/04

CPC classification number: H04N7/0135 ,  G06N3/04 ,  G06N3/08 ,  G06T5/002 ,  G06T5/003 ,  G06T5/20 ,  G06T5/50 ,  G06T2207/10016 ,  G06T2207/20004 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/20212

Abstract: Embodiments are disclosed for video image interpolation. In some embodiments, video image interpolation includes receiving a pair of input images from a digital video, determining, using a neural network, a plurality of spatially varying kernels each corresponding to a pixel of an output image, convolving a first set of spatially varying kernels with a first input image from the pair of input images and a second set of spatially varying kernels with a second input image from the pair of input images to generate filtered images, and generating the output image by performing kernel normalization on the filtered images.

公开(公告)号：US20220284613A1

公开(公告)日：2022-09-08

申请号：US17186436

申请日：2021-02-26

Applicant: Adobe Inc.

Inventor： Wei Yin ,  Jianming Zhang ,  Oliver Wang ,  Simon Niklaus ,  Mai Long ,  Su Chen

IPC: G06T7/593 ,  G06T7/30 ,  G06T7/13 ,  G06T7/143 ,  G01S17/894 ,  G01S17/42

Abstract: This disclosure describes one or more implementations of a depth prediction system that generates accurate depth images from single input digital images. In one or more implementations, the depth prediction system enforces different sets of loss functions across mix-data sources to generate a multi-branch architecture depth prediction model. For instance, in one or more implementations, the depth prediction model utilizes different data sources having different granularities of ground truth depth data to robustly train a depth prediction model. Further, given the different ground truth depth data granularities from the different data sources, the depth prediction model enforces different combinations of loss functions including an image-level normalized regression loss function and/or a pair-wise normal loss among other loss functions.

公开(公告)号：US20200334894A1

公开(公告)日：2020-10-22

申请号：US16388187

申请日：2019-04-18

Applicant: Adobe Inc.

Inventor： MAI LONG ,  Simon Niklaus ,  Jimei Yang

IPC: G06T15/20 ,  G06T7/10 ,  G06N3/04

Abstract: Systems and methods are described for generating a three dimensional (3D) effect from a two dimensional (2D) image. The methods may include generating a depth map based on a 2D image, identifying a camera path, generating one or more extremal views based on the 2D image and the camera path, generating a global point cloud by inpainting occlusion gaps in the one or more extremal views, generating one or more intermediate views based on the global point cloud and the camera path, and combining the one or more extremal views and the one or more intermediate views to produce a 3D motion effect.