公开(公告)号：US20220012885A1

公开(公告)日：2022-01-13

申请号：US17483280

申请日：2021-09-23

申请人： Adobe Inc.

发明人： Zhe Lin ,  Jianming Zhang ,  He Zhang ,  Federico Perazzi

IPC分类号： G06T7/10 ,  G06N3/04 ,  G06N3/08 ,  G06T11/60

摘要： The present disclosure relates to utilizing a neural network having a two-stream encoder architecture to accurately generate composite digital images that realistically portray a foreground object from one digital image against a scene from another digital image. For example, the disclosed systems can utilize a foreground encoder of the neural network to identify features from a foreground image and further utilize a background encoder to identify features from a background image. The disclosed systems can then utilize a decoder to fuse the features together and generate a composite digital image. The disclosed systems can train the neural network utilizing an easy-to-hard data augmentation scheme implemented via self-teaching. The disclosed systems can further incorporate the neural network within an end-to-end framework for automation of the image composition process.

公开(公告)号：US20210027470A1

公开(公告)日：2021-01-28

申请号：US16523465

申请日：2019-07-26

申请人： Adobe Inc.

发明人： Zhe Lin ,  Jianming Zhang ,  He Zhang ,  Federico Perazzi

IPC分类号： G06T7/10 ,  G06N3/04 ,  G06N3/08 ,  G06T11/60

摘要： The present disclosure relates to utilizing a neural network having a two-stream encoder architecture to accurately generate composite digital images that realistically portray a foreground object from one digital image against a scene from another digital image. For example, the disclosed systems can utilize a foreground encoder of the neural network to identify features from a foreground image and further utilize a background encoder to identify features from a background image. The disclosed systems can then utilize a decoder to fuse the features together and generate a composite digital image. The disclosed systems can train the neural network utilizing an easy-to-hard data augmentation scheme implemented via self-teaching. The disclosed systems can further incorporate the neural network within an end-to-end framework for automation of the image composition process.

公开(公告)号：US20220148135A1

公开(公告)日：2022-05-12

申请号：US17093852

申请日：2020-11-10

申请人： Adobe Inc.

发明人： Mustafa Isik ,  Michael Yanis Gharbi ,  Matthew David Fisher ,  Krishna Bhargava Mullia Lakshminarayana ,  Jonathan Eisenmann ,  Federico Perazzi

IPC分类号： G06T5/00 ,  G06N20/00 ,  G06T3/40 ,  G06K9/46 ,  G06N3/04

摘要： A plurality of pixel-based sampling points are identified within an image, wherein sampling points of a pixel are distributed within the pixel. For individual sampling points of individual pixels, a corresponding radiance vector is estimated. A radiance vector includes one or more radiance values characterizing light received at a sampling point. A first machine learning module generates, for each pixel, a corresponding intermediate radiance feature vector, based on the radiance vectors associated with the sampling points within that pixel. A second machine learning module generates, for each pixel, a corresponding final radiance feature vector, based on an intermediate radiance feature vector for that pixel, and one or more other intermediate radiance feature vectors for one or more other pixels neighboring that pixel. One or more kernels are generated, based on the final radiance feature vectors, and applied to corresponding pixels of the image, to generate a lower noise image.

公开(公告)号：US20210319232A1

公开(公告)日：2021-10-14

申请号：US16846544

申请日：2020-04-13

申请人： Adobe Inc

发明人： Federico Perazzi ,  Zhe Lin ,  Ping Hu ,  Oliver Wang ,  Fabian David Caba Heilbron

IPC分类号： G06K9/00 ,  G06N3/04 ,  G06F17/15 ,  G06T7/11

摘要： A Video Semantic Segmentation System (VSSS) is disclosed that performs accurate and fast semantic segmentation of videos using a set of temporally distributed neural networks. The VSSS receives as input a video signal comprising a contiguous sequence of temporally-related video frames. The VSSS extracts features from the video frames in the contiguous sequence and based upon the extracted features, selects, from a set of labels, a label to be associated with each pixel of each video frame in the video signal. In certain embodiments, a set of multiple neural networks are used to extract the features to be used for video segmentation and the extraction of features is distributed among the multiple neural networks in the set. A strong feature representation representing the entirety of the features is produced for each video frame in the sequence of video frames by aggregating the output features extracted by the multiple neural networks.

公开(公告)号：US20210241111A1

公开(公告)日：2021-08-05

申请号：US16782793

申请日：2020-02-05

申请人： Adobe Inc.

发明人： Shikun Liu ,  Zhe Lin ,  Yilin Wang ,  Jianming Zhang ,  Federico Perazzi

IPC分类号： G06N3/08 ,  G06N3/04

摘要： The present disclosure relates to shaping the architecture of a neural network. For example, the disclosed systems can provide a neural network shaping mechanism for at least one sampling layer of a neural network. The neural network shaping mechanism can include a learnable scaling factor between a sampling rate of the at least one sampling layer and an additional sampling function. The disclosed systems can learn the scaling factor based on a dataset while jointly learning the network weights of the neural network. Based on the learned scaling factor, the disclosed systems can shape the architecture of the neural network by modifying the sampling rate of the at least one sampling layer.

公开(公告)号：US11783184B2

公开(公告)日：2023-10-10

申请号：US17590995

申请日：2022-02-02

申请人： Adobe Inc.

发明人： Federico Perazzi ,  Zhihao Xia ,  Michael Gharbi ,  Kalyan Sunkavalli

IPC分类号： G06N3/08 ,  G06T15/50 ,  G06T5/00 ,  G06N20/10

CPC分类号： G06N3/08 ,  G06N20/10 ,  G06T5/002 ,  G06T15/50

摘要： Certain embodiments involve techniques for efficiently estimating denoising kernels for generating denoised images. For instance, a neural network receives a noisy reference image to denoise. The neural network uses a kernel dictionary of base kernels and generates a coefficient vector for each pixel in the reference image such that the coefficient vector includes a coefficient value for each base kernel in the kernel dictionary, where the base kernels are combined to generate a denoising kernel and each coefficient value indicates a contribution of a given base kernel to a denoising kernel. The neural network calculates the denoising kernel for a given pixel by applying the coefficient vector for that pixel to the kernel dictionary. The neural network applies each denoising kernel to the respective pixel to generate a denoised output image.

公开(公告)号：US11568544B2

公开(公告)日：2023-01-31

申请号：US17483280

申请日：2021-09-23

申请人： Adobe Inc.

发明人： Zhe Lin ,  Jianming Zhang ,  He Zhang ,  Federico Perazzi

IPC分类号： G06K9/62 ,  G06T7/10 ,  G06N3/04 ,  G06N3/08 ,  G06T11/60

摘要： The present disclosure relates to utilizing a neural network having a two-stream encoder architecture to accurately generate composite digital images that realistically portray a foreground object from one digital image against a scene from another digital image. For example, the disclosed systems can utilize a foreground encoder of the neural network to identify features from a foreground image and further utilize a background encoder to identify features from a background image. The disclosed systems can then utilize a decoder to fuse the features together and generate a composite digital image. The disclosed systems can train the neural network utilizing an easy-to-hard data augmentation scheme implemented via self-teaching. The disclosed systems can further incorporate the neural network within an end-to-end framework for automation of the image composition process.

公开(公告)号：US11354906B2

公开(公告)日：2022-06-07

申请号：US16846544

申请日：2020-04-13

申请人： Adobe Inc.

发明人： Federico Perazzi ,  Zhe Lin ,  Ping Hu ,  Oliver Wang ,  Fabian David Caba Heilbron

IPC分类号： G06V20/40 ,  G06N3/04 ,  G06T7/11 ,  G06F17/15

摘要： A Video Semantic Segmentation System (VSSS) is disclosed that performs accurate and fast semantic segmentation of videos using a set of temporally distributed neural networks. The VSSS receives as input a video signal comprising a contiguous sequence of temporally-related video frames. The VSSS extracts features from the video frames in the contiguous sequence and based upon the extracted features, selects, from a set of labels, a label to be associated with each pixel of each video frame in the video signal. In certain embodiments, a set of multiple neural networks are used to extract the features to be used for video segmentation and the extraction of features is distributed among the multiple neural networks in the set. A strong feature representation representing the entirety of the features is produced for each video frame in the sequence of video frames by aggregating the output features extracted by the multiple neural networks.

公开(公告)号：US20210264278A1

公开(公告)日：2021-08-26

申请号：US16799191

申请日：2020-02-24

申请人： Adobe Inc.

发明人： Shikun Liu ,  Zhe Lin ,  Yilin Wang ,  Jianming Zhang ,  Federico Perazzi

IPC分类号： G06N3/08 ,  G06N3/04

摘要： The disclosure describes one or more implementations of a neural network architecture pruning system that automatically and progressively prunes neural networks. For instance, the neural network architecture pruning system can automatically reduce the size of an untrained or previously-trained neural network without reducing the accuracy of the neural network. For example, the neural network architecture pruning system jointly trains portions of a neural network while progressively pruning redundant subsets of the neural network at each training iteration. In many instances, the neural network architecture pruning system increases the accuracy of the neural network by progressively removing excess or redundant portions (e.g., channels or layers) of the neural network. Further, by removing portions of a neural network, the neural network architecture pruning system can increase the efficiency of the neural network.

公开(公告)号：US20210150333A1

公开(公告)日：2021-05-20

申请号：US16686978

申请日：2019-11-18

申请人： Adobe Inc.

发明人： Federico Perazzi ,  Zhihao Xia ,  Michael Gharbi ,  Kalyan Sunkavalli

IPC分类号： G06N3/08 ,  G06N20/10 ,  G06T5/00 ,  G06T15/50

摘要： Certain embodiments involve techniques for efficiently estimating denoising kernels for generating denoised images. For instance, a neural network receives a noisy reference image to denoise. The neural network uses a kernel dictionary of base kernels and generates a coefficient vector for each pixel in the reference image such that the coefficient vector includes a coefficient value for each base kernel in the kernel dictionary, where the base kernels are combined to generate a denoising kernel and each coefficient value indicates a contribution of a given base kernel to a denoising kernel. The neural network calculates the denoising kernel for a given pixel by applying the coefficient vector for that pixel to the kernel dictionary. The neural network applies each denoising kernel to the respective pixel to generate a denoised output image.