A method to process applicable to coherent optical channels with either linear or nonlinear phase noise includes: splitting a received sequence of data into clusters of points according to a cumulative log-likelihood function from constellation obtained in a previous iteration; generating new constellation points by calculating a center of mass of the clusters of points; repeating until convergence or until a predetermined number of iterations has been reached to determine a signal constellation; and transmitting signals over the coherent optical channels with nonlinear phase noise using the disclosed signal constellation and LDPC-coded modulation concepts.