An apparatus for and a method of adaptively processing video signals based on noise states. The method includes, a memory device that stores a look-up table in which optimal filter tap coefficients corresponding to a number of noise levels are registered; a noise measurement unit that obtains a noise measurement from input video signals during a predetermined period; a filter coefficient determination unit that selects optimal filter tap coefficients from the memory based on the noise measurement obtained by the noise measurement unit; a noise suppression filter that employs the optimal filter tap coefficients selected by the filter coefficient determination unit to filter and remove noise from the input video signals; and a video signal processor that performs video signal processing on the result output by the noise suppression filter.