A system and method that implement a butterfly operation for a fast fourier transform operation in a processor using a matrix-vector-multiply instruction. A first set of inputs to the butterfly operation are defined as r1+j i1 and r2+j i2, and a twiddle factor Wn is defined as Wn=e−j2&pgr;/N=cos(2&pgr;/N)−j sin(2&pgr;/N)=a+jb. The butterfly operation stores r1, i1, r2 and i2 in a first set of registers and stores the twiddle factor in matrix registers. The matrix-vector-multiply instruction is executed between the matrix registers and the first set of registers.