公开(公告)号：US20210134312A1

公开(公告)日：2021-05-06

申请号：US16783021

申请日：2020-02-05

Applicant: Microsoft Technology Licensing, LLC

Inventor： Kazuhito KOISHIDA ,  Michael IUZZOLINO

IPC: G10L21/02 ,  G06K9/00 ,  G10L15/25 ,  G10L25/18 ,  G10L15/22

Abstract: Example speech enhancement systems include a spatio-temporal residual network configured to receive video data containing a target speaker and extract visual features from the video data, an autoencoder configured to receive input of an audio spectrogram and extract audio features from the audio spectrogram, and a squeeze-excitation fusion block configured to receive input of visual features from a layer of the spatio-temporal residual network and input of audio features from a layer of the autoencoder, and to provide an output to the decoder of the autoencoder. The decoder is configured to output a mask configured based upon the fusion of audio features and visual features by the squeeze-excitation fusion block, and the instructions are executable to apply the mask to the audio spectrogram to generate an enhanced magnitude spectrogram, and to reconstruct an enhanced waveform from the enhanced magnitude spectrogram.