公开(公告)号：US20220122221A1

公开(公告)日：2022-04-21

申请号：US17384357

申请日：2021-07-23

Applicant: Adobe Inc.

Inventor： Cameron Smith ,  Ratheesh Kalarot ,  Wei-An Lin ,  Richard Zhang ,  Niloy Mitra ,  Elya Shechtman ,  Shabnam Ghadar ,  Zhixin Shu ,  Yannick Hold-Geoffrey ,  Nathan Carr ,  Jingwan Lu ,  Oliver Wang ,  Jun-Yan Zhu

IPC: G06T3/40 ,  G06F3/0484 ,  G06N3/08 ,  G06N3/04

Abstract: An improved system architecture uses a pipeline including a Generative Adversarial Network (GAN) including a generator neural network and a discriminator neural network to generate an image. An input image in a first domain and information about a target domain are obtained. The domains correspond to image styles. An initial latent space representation of the input image is produced by encoding the input image. An initial output image is generated by processing the initial latent space representation with the generator neural network. Using the discriminator neural network, a score is computed indicating whether the initial output image is in the target domain. A loss is computed based on the computed score. The loss is minimized to compute an updated latent space representation. The updated latent space representation is processed with the generator neural network to generate an output image in the target domain.

公开(公告)号：US20220121931A1

公开(公告)日：2022-04-21

申请号：US17384371

申请日：2021-07-23

Applicant: Adobe Inc.

Inventor： Ratheesh Kalarot ,  Wei-An Lin ,  Cameron Smith ,  Zhixin Shu ,  Baldo Faieta ,  Shabnam Ghadar ,  Jingwan Lu ,  Aliakbar Darabi ,  Jun-Yan Zhu ,  Niloy Mitra ,  Richard Zhang ,  Elya Shechtman

IPC: G06N3/08 ,  G06N3/04 ,  G06K9/62

Abstract: Systems and methods train and apply a specialized encoder neural network for fast and accurate projection into the latent space of a Generative Adversarial Network (GAN). The specialized encoder neural network includes an input layer, a feature extraction layer, and a bottleneck layer positioned after the feature extraction layer. The projection process includes providing an input image to the encoder and producing, by the encoder, a latent space representation of the input image. Producing the latent space representation includes extracting a feature vector from the feature extraction layer, providing the feature vector to the bottleneck layer as input, and producing the latent space representation as output. The latent space representation produced by the encoder is provided as input to the GAN, which generates an output image based upon the latent space representation. The encoder is trained using specialized loss functions including a segmentation loss and a mean latent loss.

公开(公告)号：US11189094B2

公开(公告)日：2021-11-30

申请号：US16985402

申请日：2020-08-05

Applicant: Adobe, Inc.

Inventor： Oliver Wang ,  Vladimir Kim ,  Matthew Fisher ,  Elya Shechtman ,  Chen-Hsuan Lin ,  Bryan Russell

IPC: G06T17/20 ,  G06T15/00 ,  G06N3/08 ,  B29C64/386 ,  B33Y50/00 ,  G06T7/55

Abstract: Techniques are disclosed for 3D object reconstruction using photometric mesh representations. A decoder is pretrained to transform points sampled from 2D patches of representative objects into 3D polygonal meshes. An image frame of the object is fed into an encoder to get an initial latent code vector. For each frame and camera pair from the sequence, a polygonal mesh is rendered at the given viewpoints. The mesh is optimized by creating a virtual viewpoint, rasterized to obtain a depth map. The 3D mesh projections are aligned by projecting the coordinates corresponding to the polygonal face vertices of the rasterized mesh to both selected viewpoints. The photometric error is determined from RGB pixel intensities sampled from both frames. Gradients from the photometric error are backpropagated into the vertices of the assigned polygonal indices by relating the barycentric coordinates of each image to update the latent code vector.

公开(公告)号：US20210357684A1

公开(公告)日：2021-11-18

申请号：US15930539

申请日：2020-05-13

Applicant: Adobe Inc.

Inventor： Sohrab Amirghodsi ,  Zhe Lin ,  Yilin Wang ,  Tianshu Yu ,  Connelly Barnes ,  Elya Shechtman

IPC: G06K9/62 ,  G06N3/08 ,  G06N20/00 ,  G06F17/18

Abstract: A panoptic labeling system includes a modified panoptic labeling neural network (“modified PLNN”) that is trained to generate labels for pixels in an input image. The panoptic labeling system generates modified training images by combining training images with mask instances from annotated images. The modified PLNN determines a set of labels representing categories of objects depicted in the modified training images. The modified PLNN also determines a subset of the labels representing categories of objects depicted in the input image. For each mask pixel in a modified training image, the modified PLNN calculates a probability indicating whether the mask pixel has the same label as an object pixel. The modified PLNN generates a mask label for each mask pixel, based on the probability. The panoptic labeling system provides the mask label to, for example, a digital graphics editing system that uses the labels to complete an infill operation.

公开(公告)号：US11080833B2

公开(公告)日：2021-08-03

申请号：US16692843

申请日：2019-11-22

Applicant: Adobe Inc.

Inventor： Connelly Barnes ,  Utkarsh Singhal ,  Elya Shechtman ,  Michael Gharbi

IPC: G06T5/50 ,  G06K9/62 ,  G06T5/00 ,  G06N3/04 ,  G06F16/53 ,  G06N3/08

Abstract: A method for manipulating a target image includes generating a query of the target image and keys and values of a first reference image. The method also includes generating matching costs by comparing the query of the target image with each key of the reference image and generating a set of weights from the matching costs. Further, the method includes generating a set of weighted values by applying each weight of the set of weights to a corresponding value of the values of the reference image and generating a weighted patch by adding each weighted value of the set of weighted values together. Additionally, the method includes generating a combined weighted patch by combining the weighted patch with additional weighted patches associated with additional queries of the target image and generating a manipulated image by applying the combined weighted patch to an image processing algorithm.

公开(公告)号：US20210233213A1

公开(公告)日：2021-07-29

申请号：US16752030

申请日：2020-01-24

Applicant: Adobe Inc.

Inventor： Youssef Alami Mejjati ,  Zoya Bylinskii ,  Elya Shechtman

IPC: G06T5/00 ,  G06T7/194

Abstract: Techniques of adjusting the salience of an image include generating values of photographic development parameters for a foreground and background of an image to adjust the salience of the image in the foreground. These parameters are global in nature over the image rather than local. Moreover, the optimization of the salience over such sets of global parameters is provided through two sets of these parameters by an encoder: one set corresponding to the foreground, in which the salience is to be either increased or decreased, and the other set corresponding to the background. Once the set of development parameters corresponding to the foreground region and the set of development parameters corresponding to the background region have been determined, a decoder generates an adjusted image with an increased salience based on these sets of development parameters.

公开(公告)号：US20200372619A1

公开(公告)日：2020-11-26

申请号：US16420782

申请日：2019-05-23

Applicant: ADOBE INC.

Inventor： Sohrab Amirghodsi ,  Elya Shechtman ,  Derek Novo

IPC: G06T5/00 ,  G06T7/194 ,  G06T7/90 ,  G06T7/70

Abstract: Embodiments of the present invention provide systems, methods, and computer storage media for automatically synthesizing a content-aware sampling region for a hole-filling algorithm such as content-aware fill. Given a source image and a hole (or other target region to fill), a sampling region can be synthesized by identifying a band of pixels surrounding the hole, clustering these pixels based on one or more characteristics (e.g., color, x/y coordinates, depth, focus, etc.), passing each of the resulting clusters as foreground pixels to a segmentation algorithm, and unioning the resulting pixels to form the sampling region. The sampling region can be stored in a constraint mask and passed to a hole-filling algorithm such as content-aware fill to synthesize a fill for the hole (or other target region) from patches sampled from the synthesized sampling region.

公开(公告)号：US10783691B2

公开(公告)日：2020-09-22

申请号：US16681218

申请日：2019-11-12

Applicant: Adobe Inc. ,  CZECH TECHNICAL UNIVERSITY IN PRAGUE

Inventor： David Simons ,  Michal Lukac ,  Daniel Sykora ,  Elya Shechtman ,  Paul Asente ,  Jingwan Lu ,  Jakub Fiser ,  Ondrej Jamriska

IPC: G06T11/00 ,  G06T13/40 ,  G06K9/00 ,  G06T15/04 ,  G06T7/194 ,  G06T7/11 ,  G06T7/73 ,  G06T7/90 ,  G06T3/00 ,  G06T5/00 ,  G06T5/50 ,  G06T13/80 ,  G06T7/60 ,  G06T15/02

Abstract: Certain embodiments involve generating one or more of appearance guide and a positional guide and using one or more of the guides to synthesize a stylized image or animation. For example, a system obtains data indicating a target image and a style exemplar image. The system generates an appearance guide, a positional guide, or both from the target image and the style exemplar image. The system uses one or more of the guides to transfer a texture or style from the style exemplar image to the target image.

公开(公告)号：US20200265294A1

公开(公告)日：2020-08-20

申请号：US16276559

申请日：2019-02-14

Applicant: Adobe Inc.

Inventor： Vladimir Kim ,  Omid Poursaeed ,  Jun Saito ,  Elya Shechtman

IPC: G06N3/04 ,  G06T13/00 ,  G06T17/20 ,  G06N3/08

Abstract: In implementations of object animation using generative neural networks, one or more computing devices of a system implement an animation system for reproducing animation of an object in a digital video. A mesh of the object is obtained from a first frame of the digital video and a second frame of the digital video having the object is selected. Features of the object from the second frame are mapped to vertices of the mesh, and the mesh is warped based on the mapping. The warped mesh is rendered as an image by a neural renderer and compared to the object from the second frame to train a neural network. The rendered image is then refined by a generator of a generative adversarial network which includes a discriminator. The discriminator trains the generator to reproduce the object from the second frame as the refined image.

公开(公告)号：US10719742B2

公开(公告)日：2020-07-21

申请号：US15897910

申请日：2018-02-15

Applicant: Adobe Inc.

Inventor： Elya Shechtman ,  Oliver Wang ,  Mehmet Yumer ,  Chen-Hsuan Lin

IPC: G06K9/66 ,  G06N3/04 ,  G06N3/08

Abstract: The present disclosure relates to an image composite system that employs a generative adversarial network to generate realistic composite images. For example, in one or more embodiments, the image composite system trains a geometric prediction neural network using an adversarial discrimination neural network to learn warp parameters that provide correct geometric alignment of foreground objects with respect to a background image. Once trained, the determined warp parameters provide realistic geometric corrections to foreground objects such that the warped foreground objects appear to blend into background images naturally when composited together.