A system and method to reduce roundoff error of Fast Fourier transform (FFT) operation. Data which comes out as an irrational number (a square root) out of twiddle factors on a complex plane, included in a butterfly operation (8p) is preserved intentionally without being calculated in one stage of multiple stages of a multi-stage pipelined FFT, and when it occurs again in a later stage, an operation to multiply the two twiddle factors with each other is performed. This enables to eliminate roundoff errors during the butterfly operation 8p of radix-8. Other applications are also possible such as by overlaying a further stage by a butterfly operation of radix-2 or radix-4.