公开(公告)号：US20240282012A1

公开(公告)日：2024-08-22

申请号：US18442622

申请日：2024-02-15

Applicant: QUALCOMM Incorporated

Inventor： Dmytro Rusanovskyy ,  Samuel James Eadie ,  Yun Li ,  Marta Karczewicz

IPC: G06T9/00 ,  H04N19/105 ,  H04N19/176 ,  H04N19/192 ,  H04N19/70 ,  H04N19/82

CPC classification number: G06T9/002 ,  H04N19/105 ,  H04N19/176 ,  H04N19/192 ,  H04N19/70 ,  H04N19/82

Abstract: A video encoder and video decoder are configured to perform a neural network (NN)-based filter process on reconstructed blocks of video data. In one example, the NN-based filter process uses reconstruction samples of the block, prediction samples of the block, and supplementary data related to the block as inputs. The NN-based filter process includes an initial processing of one or more types of the supplementary data with fewer computations relative to the initial processing of the reconstruction samples and the prediction samples.

公开(公告)号：US20250119556A1

公开(公告)日：2025-04-10

申请号：US18889977

申请日：2024-09-19

Applicant: QUALCOMM Incorporated

Inventor： Yun Li ,  Dmytro Rusanovskyy ,  Thomas Alexander Ryder ,  Samuel James Eadie ,  Marta Karczewicz

IPC: H04N19/176 ,  G06T9/00 ,  H04N19/117 ,  H04N19/159 ,  H04N19/82

Abstract: A method of processing video data includes receiving a picture; and filtering a current block of the picture, through a neural network and based on local correlations of proximate samples and distant, non-local correlations of non-proximate samples relative to the current block, to generate a filtered current block. The neural network comprises one or more backbone blocks and one or more transformer blocks. Each of the one or more transformer blocks is associated with a backbone block of the one or more backbone blocks. At least one of the backbone blocks is configured to capture the local correlations, relative to the current block and the proximate samples of the current block, and at least one of the transformer blocks is configured to generate features, based on applying an attention mechanism, that capture the distant, non-local correlations, relative to the current block and the non-proximate samples, in the picture for processing.

公开(公告)号：US20240015312A1

公开(公告)日：2024-01-11

申请号：US18331674

申请日：2023-06-08

Applicant: QUALCOMM Incorporated

Inventor： Hongtao Wang ,  Samuel James Eadie ,  Muhammed Zeyd Coban ,  Marta Karczewicz

IPC: H04N19/186 ,  H04N19/70 ,  H04N19/42 ,  H04N19/176 ,  H04N19/80 ,  H04N19/124

CPC classification number: H04N19/186 ,  H04N19/70 ,  H04N19/42 ,  H04N19/176 ,  H04N19/80 ,  H04N19/124

Abstract: A method of processing video data includes receiving a syntax element that defines a filtering mode for a neural network (NN) model for both a first color component and a second color component, applying an instance of the NN model, in the defined filtering mode, to a first block of the first color component to generate a first filtered block, and storing the first filtered block for a coding unit (CU).

公开(公告)号：US20250016339A1

公开(公告)日：2025-01-09

申请号：US18744171

申请日：2024-06-14

Applicant: QUALCOMM Incorporated

Inventor： Thomas Alexander Ryder ,  Samuel James Eadie ,  Marta Karczewicz ,  Muhammed Zeyd Coban ,  Vadim Seregin

IPC: H04N19/31 ,  H04N19/107 ,  H04N19/167 ,  H04N19/176

Abstract: An example device for decoding video data includes a processing system comprising one or more processors implemented in circuitry and configured to: determine that a first temporal layer identifier of a first picture of the video data is included in a first set of temporal layers; in response to the first temporal layer identifier being included in the first set of temporal layers, decode blocks of the first picture on a block by block basis; determine that a second temporal layer identifier of a second picture of the video data is included in a second set of temporal layers, the second set of temporal layers being higher than the first set of temporal layers; and in response to the second temporal layer identifier being included in the second set of temporal layers, execute a neural network-based video decoder to decode the second picture.

公开(公告)号：US20240015284A1

公开(公告)日：2024-01-11

申请号：US18346620

申请日：2023-07-03

Applicant: QUALCOMM Incorporated

Inventor： Hongtao Wang ,  Samuel James Eadie ,  Muhammed Zeyd Coban ,  Marta Karczewicz

IPC: H04N19/117 ,  H04N19/80 ,  H04N19/176

CPC classification number: H04N19/117 ,  H04N19/80 ,  H04N19/176

Abstract: An example device for filtering video data includes a memory configured to store video data; and a processing system comprising one or more processors implemented in circuitry, the processing system being configured to: apply one or more neural network processing blocks to intermediate filtered video data, each of the neural network processing blocks including a first 1×1 convolutional filter, a parametric rectified linear unit (PReLU) filter, a second 1×1 convolutional filter, and a 3×3 convolutional filter; apply additional neural network processing blocks to output of the one or more neural network processing blocks to form filtered video data; and output the filtered video data.

公开(公告)号：US20250119592A1

公开(公告)日：2025-04-10

申请号：US18888423

申请日：2024-09-18

Applicant: QUALCOMM Incorporated

Inventor： Thomas Alexander Ryder ,  Dmytro Rusanovskyy ,  Samuel James Eadie ,  Yun Li ,  Marta Karczewicz

IPC: H04N19/82 ,  H04N19/117 ,  H04N19/132 ,  H04N19/159 ,  H04N19/176

Abstract: Methods and devices for decoding video data are described. An example method includes in-loop filtering a current block of the video data using a neural network-based in-loop filter to generate an in-loop filtered current block, wherein the neural network-based in-loop filter is trained using an architecture comprising a U-Net architecture comprising one or more residual blocks and one or more transform blocks; and outputting the in-loop filtered current block.

公开(公告)号：US20240348837A1

公开(公告)日：2024-10-17

申请号：US18631748

申请日：2024-04-10

Applicant: QUALCOMM Incorporated

Inventor： Dmytro Rusanovskyy ,  Yun Li ,  Samuel James Eadie ,  Marta Karczewicz

IPC: H04N19/82 ,  H04N19/117 ,  H04N19/172 ,  H04N19/176

CPC classification number: H04N19/82 ,  H04N19/117 ,  H04N19/172 ,  H04N19/176

Abstract: A device for decoding video data receives a picture of video data; reconstructs a block of the picture of video data to generate a reconstructed block; and performs a neural network (NN)-based filter process on the reconstructed block to generate a filtered block, wherein the NN-based filter process includes performing a plurality of separable convolutions in parallel with a point-wise input convolution.

公开(公告)号：US20240283925A1

公开(公告)日：2024-08-22

申请号：US18442955

申请日：2024-02-15

Applicant: QUALCOMM Incorporated

Inventor： Dmytro Rusanovskyy ,  Samuel James Eadie ,  Yun Li ,  Marta Karczewicz

IPC: H04N19/117 ,  H04N19/105 ,  H04N19/176

CPC classification number: H04N19/117 ,  H04N19/105 ,  H04N19/176

Abstract: A video coder is configured to perform a neural network (NN)-based filter process on reconstructed blocks of vide data. In one example, a video coder may receive a picture of video data, and reconstruct a block of the picture of video data to generate a reconstructed block. The video coder may perform the NN-based filter process on the reconstructed block to generate a filtered block, wherein the NN-based filter process includes performing a plurality separable convolutions to approximate a multi-dimensional convolution.