In a coherent optical receiver, a frequency domain engine digitally processes at least two multi-bit sample streams of a received optical signal. The frequency domain engine includes a Fast Fourier Transform (FFT) filter for computing a complex vector representative of a frequency-domain spectrum of the received optical signal. A transpose and add block computes a vector sum of the complex vector and a transposed version of the complex vector, and an Inverse Fast Fourier Transform (IFFT) filter computes a complex output vector from the addition result. With this arrangement, parallel real filter operations are efficiently performed on each of the multi-bit sample streams, using a single back-to-back FFT-IFFT filter structure.