Systems and methods for communication using an optical transmission system having optical transmitters and receivers includes performing a low-density parity-check (LDPC) encoding; performing nonbinary pm-ary signaling, where p is a prime larger than 2; taking m p-ary symbols at a time and selecting a point from pm-ary signal constellation; after up-sampling and driving amplification, using coordinates of the pm-ary constellation as input of I/Q modulator x (y); combining two independent pm-ary streams corresponding to x and y-polarization states by a polarization beam combiner (PBS) and transmitting data over the optical transmission system.