Disclosed herein is a signal processing apparatus and method based on multiple textures using video audio excitation signals. For this, an input signal that includes a video signal and an audio signal is divided into unit component signals, and one is selected from a plurality of frames of each unit component signal as a seed signal. A plurality of texture points are detected from the seed signal. The texture points are tracked from the frames of the unit component signal and then spatio-temporal location transform variables for the texture points are calculated. Texture signals are defined using texture points at which the spatio-temporal location transform variables correspond to one another. Each of the texture signals is defined as a sum of a plurality of texture blocks that are outputs of texture synthesis filters that receive video audio excitation signals as inputs.