公开(公告)号：US20240338915A1

公开(公告)日：2024-10-10

申请号：US18132272

申请日：2023-04-07

Applicant: Adobe Inc.

Inventor： Zhixin Shu ,  Zexiang Xu ,  Shahrukh Athar ,  Sai Bi ,  Kalyan Sunkavalli ,  Fujun Luan

IPC: G06T19/20 ,  G06N3/08 ,  G06T15/80 ,  G06T17/20

CPC classification number: G06T19/20 ,  G06N3/08 ,  G06T15/80 ,  G06T17/20 ,  G06T2210/44 ,  G06T2219/2012 ,  G06T2219/2021

Abstract: Certain aspects and features of this disclosure relate to providing a controllable, dynamic appearance for neural 3D portraits. For example, a method involves projecting a color at points in a digital video portrait based on location, surface normal, and viewing direction for each respective point in a canonical space. The method also involves projecting, using the color, dynamic face normals for the points as changing according to an articulated head pose and facial expression in the digital video portrait. The method further involves disentangling, based on the dynamic face normals, a facial appearance in the digital video portrait into intrinsic components in the canonical space. The method additionally involves storing and/or rendering at least a portion of a head pose as a controllable, neural 3D portrait based on the digital video portrait using the intrinsic components.

公开(公告)号：US20240169553A1

公开(公告)日：2024-05-23

申请号：US18057436

申请日：2022-11-21

Applicant: Adobe Inc.

Inventor： Jae shin Yoon ,  Zhixin Shu ,  Yangtuanfeng Wang ,  Jingwan Lu ,  Jimei Yang ,  Duygu Ceylan Aksit

IPC: G06T7/20 ,  G06T13/40 ,  G06T15/04 ,  G06T17/00

CPC classification number: G06T7/20 ,  G06T13/40 ,  G06T15/04 ,  G06T17/00 ,  G06T2207/10016 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/30244

Abstract: Techniques for modeling secondary motion based on three-dimensional models are described as implemented by a secondary motion modeling system, which is configured to receive a plurality of three-dimensional object models representing an object. Based on the three-dimensional object models, the secondary motion modeling system determines three-dimensional motion descriptors of a particular three-dimensional object model using one or more machine learning models. Based on the three-dimensional motion descriptors, the secondary motion modeling system models at least one feature subjected to secondary motion using the one or more machine learning models. The particular three-dimensional object model having the at least one feature is rendered by the secondary motion modeling system.

公开(公告)号：US20220122305A1

公开(公告)日：2022-04-21

申请号：US17384273

申请日：2021-07-23

Applicant: Adobe Inc.

Inventor： Cameron Smith ,  Ratheesh Kalarot ,  Wei-An Lin ,  Richard Zhang ,  Niloy Mitra ,  Elya Shechtman ,  Shabnam Ghadar ,  Zhixin Shu ,  Yannick Hold-Geoffrey ,  Nathan Carr ,  Jingwan Lu ,  Oliver Wang ,  Jun-Yan Zhu

IPC: G06T11/60 ,  G06T3/40

Abstract: An improved system architecture uses a pipeline including an encoder and a Generative Adversarial Network (GAN) including a generator neural network to generate edited images with improved speed, realism, and identity preservation. The encoder produces an initial latent space representation of an input image by encoding the input image. The generator neural network generates an initial output image by processing the initial latent space representation of the input image. The system generates an optimized latent space representation of the input image using a loss minimization technique that minimizes a loss between the input image and the initial output image. The loss is based on target perceptual features extracted from the input image and initial perceptual features extracted from the initial output image. The system outputs the optimized latent space representation of the input image for downstream use.

公开(公告)号：US20220122222A1

公开(公告)日：2022-04-21

申请号：US17384283

申请日：2021-07-23

Applicant: Adobe Inc.

Inventor： Cameron Smith ,  Ratheesh Kalarot ,  Wei-An Lin ,  Richard Zhang ,  Niloy Mitra ,  Elya Shechtman ,  Shabnam Ghadar ,  Zhixin Shu ,  Yannick Hold-Geoffrey ,  Nathan Carr ,  Jingwan Lu ,  Oliver Wang ,  Jun-Yan Zhu

IPC: G06T3/40 ,  G06T11/60

Abstract: An improved system architecture uses a Generative Adversarial Network (GAN) including a specialized generator neural network to generate multiple resolution output images. The system produces a latent space representation of an input image. The system generates a first output image at a first resolution by providing the latent space representation of the input image as input to a generator neural network comprising an input layer, an output layer, and a plurality of intermediate layers and taking the first output image from an intermediate layer, of the plurality of intermediate layers of the generator neural network. The system generates a second output image at a second resolution different from the first resolution by providing the latent space representation of the input image as input to the generator neural network and taking the second output image from the output layer of the generator neural network.

公开(公告)号：US20220121932A1

公开(公告)日：2022-04-21

申请号：US17384378

申请日：2021-07-23

Applicant: Adobe Inc.

Inventor： Ratheesh Kalarot ,  Wei-An Lin ,  Cameron Smith ,  Zhixin Shu ,  Baldo Faieta ,  Shabnam Ghadar ,  Jingwan Lu ,  Aliakbar Darabi ,  Jun-Yan Zhu ,  Niloy Mitra ,  Richard Zhang ,  Elya Shechtman

IPC: G06N3/08 ,  G06N3/04

Abstract: Systems and methods train an encoder neural network for fast and accurate projection into the latent space of a Generative Adversarial Network (GAN). The encoder is trained by providing an input training image to the encoder and producing, by the encoder, a latent space representation of the input training image. The latent space representation is provided as input to the GAN to generate a generated training image. A latent code is sampled from a latent space associated with the GAN and the sampled latent code is provided as input to the GAN. The GAN generates a synthetic training image based on the sampled latent code. The sampled latent code is provided as input to the encoder to produce a synthetic training code. The encoder is updated by minimizing a loss between the generated training image and the input training image, and the synthetic training code and the sampled latent code.

公开(公告)号：US10521892B2

公开(公告)日：2019-12-31

申请号：US15253655

申请日：2016-08-31

Applicant: ADOBE INC.

Inventor： Kalyan K. Sunkavalli ,  Sunil Hadap ,  Elya Shechtman ,  Zhixin Shu

IPC: G06T5/40 ,  G06K9/00 ,  G06T5/50 ,  G06T11/00

Abstract: Embodiments of the present invention provide systems, methods, and computer storage media directed at relighting a target image based on a lighting effect from a reference image. In one embodiment, a target image and a reference image are received, the reference image includes a lighting effect desired to be applied to the target image. A lighting transfer is performed using color data and geometrical data associated with the reference image and color data and geometrical data associated with the target image. The lighting transfer causes generation of a relit image that corresponds with the target image having a lighting effect of the reference image. The relit image is provided for display to a user via one or more output devices. Other embodiments may be described and/or claimed.

公开(公告)号：US20240233318A9

公开(公告)日：2024-07-11

申请号：US17971169

申请日：2022-10-21

Applicant: Adobe Inc.

Inventor： Yijun Li ,  Zhixin Shu ,  Zhen Zhu ,  Krishna Kumar Singh

IPC: G06V10/70 ,  G06N3/04 ,  G06T11/00 ,  G06T15/08

CPC classification number: G06V10/70 ,  G06N3/0454 ,  G06T11/001 ,  G06T15/08

Abstract: An image generation system implements a multi-branch GAN to generate images that each express visually similar content in a different modality. A generator portion of the multi-branch GAN includes multiple branches that are each tasked with generating one of the different modalities. A discriminator portion of the multi-branch GAN includes multiple fidelity discriminators, one for each of the generator branches, and a consistency discriminator, which constrains the outputs generated by the different generator branches to appear visually similar to one another. During training, outputs from each of the fidelity discriminators and the consistency discriminator are used to compute a non-saturating GAN loss. The non-saturating GAN loss is used to refine parameters of the multi-branch GAN during training until model convergence. The trained multi-branch GAN generates multiple images from a single input, where each of the multiple images depicts visually similar content expressed in a different modality.

公开(公告)号：US20230316591A1

公开(公告)日：2023-10-05

申请号：US17709895

申请日：2022-03-31

Applicant: Adobe Inc.

Inventor： Zhixin Shu ,  Zhe Lin ,  Yuchen Liu ,  Yijun Li ,  Richard Zhang

IPC: G06T11/00 ,  G06V10/40 ,  G06V10/774

CPC classification number: G06T11/00 ,  G06V10/40 ,  G06V10/7747

Abstract: Techniques for identity preserved controllable facial image manipulation are described that support generation of a manipulated digital image based on a facial image and a render image. For instance, a facial image depicting a facial representation of an individual is received as input. A feature space including an identity parameter and at least one other visual parameter is extracted from the facial image. An editing module edits one or more of the visual parameters and preserves the identity parameter. A renderer generates a render image depicting a morphable model reconstruction of the facial image based on the edit. The render image and facial image are encoded, and a generator of a neural network is implemented to generate a manipulated digital image based on the encoded facial image and the encoded render image.

公开(公告)号：US11748928B2

公开(公告)日：2023-09-05

申请号：US17094093

申请日：2020-11-10

Applicant: Adobe Inc.

Inventor： Yang Yang ,  Zhixin Shu ,  Shabnam Ghadar ,  Jingwan Lu ,  Jakub Fiser ,  Elya Schechtman ,  Cameron Y. Smith ,  Baldo Antonio Faieta ,  Alex Charles Filipkowski

IPC: G06T11/60 ,  G06F21/62 ,  G06F16/56 ,  G06F16/532

CPC classification number: G06T11/60 ,  G06F16/532 ,  G06F16/56 ,  G06F21/6254 ,  G06T2200/24

Abstract: Face anonymization techniques are described that overcome conventional challenges to generate an anonymized face. In one example, a digital object editing system is configured to generate an anonymized face based on a target face and a reference face. As part of this, the digital object editing system employs an encoder as part of machine learning to extract a target encoding of the target face image and a reference encoding of the reference face. The digital object editing system then generates a mixed encoding from the target and reference encodings. The mixed encoding is employed by a machine-learning model of the digital object editing system to generate a mixed face. An object replacement module is used by the digital object editing system to replace the target face in the target digital image with the mixed face.

公开(公告)号：US20220122306A1

公开(公告)日：2022-04-21

申请号：US17468487

申请日：2021-09-07

Applicant: Adobe Inc.

Inventor： Wei-An Lin ,  Baldo Faieta ,  Cameron Smith ,  Elya Shechtman ,  Jingwan Lu ,  Jun-Yan Zhu ,  Niloy Mitra ,  Ratheesh Kalarot ,  Richard Zhang ,  Shabnam Ghadar ,  Zhixin Shu

IPC: G06T11/60 ,  G06F3/0484 ,  G06N3/08 ,  G06N3/04

Abstract: Systems and methods dynamically adjust an available range for editing an attribute in an image. An image editing system computes a metric for an attribute in an input image as a function of a latent space representation of the input image and a filtering vector for editing the input image. The image editing system compares the metric to a threshold. If the metric exceeds the threshold, then the image editing system selects a first range for editing the attribute in the input image. If the metric does not exceed the threshold, a second range is selected. The image editing system causes display of a user interface for editing the input image comprising an interface element for editing the attribute within the selected range.