Adaptive filtering techniques are set forth that improve video playback performance. A color similarity metric is computed for source pixels undergoing an image transform operation to determine color values of a target pixel location. The value of the color similarity metric defines what algorithm is used to determine the color values of the target pixel location from the source pixels. While the computation of the color similarity metric introduces additional instructions for each pixel, going against conventional wisdom, enormous performance savings by avoiding unnecessarily complex sets of interpolation instructions where a simpler instruction set suffices, outweighs any such cost from additional instructions.