An edge adaptive system and method for image filtering. The method maps each output pixel onto input image coordinates and then prefilters and resamples the input image pixels around this point to reduce noise and adjust the scale corresponding to a particular operation. Then the edge in the input image is detected based on local and average signal variances in the input pixels. According to the edge detection parameters, including orientation, anisotropy and variance strength, the method determines a footprint and frequency response for the interpolation of the output pixel. In a more particular implementation, the method divides the input pixel space into a finite number of directions called skews, and estimates the edge orientation with the nearest skew direction. This further facilitates pixels inclusion in the interpolation of the output pixel.