公开(公告)号：US11836952B2

公开(公告)日：2023-12-05

申请号：US17240510

申请日：2021-04-26

Applicant: Microsoft Technology Licensing, LLC

Inventor： Sunando Sengupta ,  Alexandros Neofytou ,  Eric Chris Wolfgang Sommerlade ,  Yang Liu

IPC: G06K9/00 ,  G06T9/00 ,  G06T3/60 ,  G10L19/012 ,  G10L25/51 ,  G06F18/21 ,  G10L19/00

CPC classification number: G06T9/00 ,  G06F18/21 ,  G06T3/60 ,  G10L19/012 ,  G10L25/51 ,  G10L2019/0002

Abstract: In various embodiments, a computer-implemented method of training a neural network for creating an output signal of different modality from an input signal is described. In embodiments, the first modality may be a sound signal or a visual image and where the output signal would be a visual image or a sound signal, respectively. In embodiments a model is trained using a first pair of visual and audio networks to train a set of codebooks using known visual signals and the audio signals and using a second pair of visual and audio networks to further train the set of codebooks using the augmented visual signals and the augmented audio signals. Further, the first and the second visual networks are equally weighted and where the first and the second audio networks are equally weighted. In aspects of the present disclosure, the set of codebooks comprise a visual codebook, an audio codebook and a correlation codebook. These codebooks are then used to create an visual image from a sound signal and/or a sound signal from a visual image.

公开(公告)号：US11714881B2

公开(公告)日：2023-08-01

申请号：US17331876

申请日：2021-05-27

Applicant: Microsoft Technology Licensing, LLC

Inventor： Eric Chris Wolfgang Sommerlade ,  Sunando Sengupta ,  Alexandros Neophytou

IPC: G06K9/00 ,  G06F18/24 ,  G06T7/194 ,  G06N3/04 ,  G06T3/40 ,  G06T5/50 ,  G06F18/2413

CPC classification number: G06F18/24765 ,  G06F18/2413 ,  G06N3/04 ,  G06T3/4053 ,  G06T5/50 ,  G06T7/194 ,  G06T2207/20221 ,  G06T2207/30196

Abstract: A method of improving image quality of a stream of input images is described. The stream of input images, including a current input image, is received. One or more target objects, including a first target object, are identified spatio-temporally within the stream of input images. The one or more target objects are tracked spatio-temporally within the stream of input images. The current input image is segmented into i) a foreground including the first target object, and ii) a background. The foreground is processed to have improved image quality in the current input image. Processing of the foreground further comprises processing the first target object using a same processing technique as for a prior input image of the stream of input images based on the tracking of the first target object. The background is processed differently from the foreground. An output image is generated by merging the foreground with the background.