Modulating method and apparatus in a broadcasting and communication system are provided. The modulating apparatus includes: a first quadrature phase shift keying (QPSK) modulating unit QPSK-modulating first data including in-phase (I) data and quadrature (Q) data to generate a first signal; a phase rotating unit rotating a phase of the first signal by θ to generate a phase rotated signal; a second QPSK modulating unit QPSK-modulating second data different from the first data and including I′ data and Q′ data to generate a second signal; and a layer modulating unit modulating the phase rotated signal and the second signal to output a final signal.