An echo canceller includes an adaptive digital filter that generates an estimated echo signal z[k] in response to (i) a sampled input data sequence x[k] and (ii) an error signal sequence e[k] indicative of the difference between a far end signal sequence y[k] and the estimated echo signal z[k]. The adaptive filter includes N filter taps that each provide an associated tap output signal, wherein the adaptive digital filter generates the estimated echo signal z[k] using the associated tap output signals from M of the N filter taps selected in response to a time delay estimate signal. The adaptive filter computes filter coefficients for each of the M number of the N filter taps using the associated tap output signals from the M number of said N filter taps. The echo canceller also includes atime delay estimator that is responsive to the sampled input data sequence x[k] and the signal sequence y[k], and estimates a plurality of delays within the sequence x[k], and provides a time delay estimate that is indicative of the location of the plurality of delays within the sequence x[k]. A summer computes the difference between signal sequence y[k] and the estimated echo signal z[k] and provides an output signal indicative thereof. The echo canceller of the present invention can be considered a sparse echo canceller, since the adaptive filter selectively uses a subset of the available filter taps to compute the estimated echo signal. The filter taps are selected based upon time delay estimate data associated with the echo. For example, an adaptive filter having N taps may process signals from M of the N filter taps, where the M taps are selected based upon the time delay estimation from the time delay estimator.