Disclosed herein is a video processing apparatus and method based on multiple texture images using video excitation signals. For this, an input video is divided into shot segment, and one is selected from a plurality of frames of each shot segment as a seed image. A plurality of texture points are detected from the seed image. The plurality of texture points are tracked from the plurality of frames of the shot segment and then spatio-temporal location transform variables for the respective texture points are calculated. A plurality of texture images are defined using texture points at which the spatio-temporal location transform variables correspond to one another. Each of the plurality of texture images is defined as a sum of a plurality of texture blocks that are outputs of texture synthesis filters that receive video excitation signals as inputs.