A vestigial sideband (VSB) modulation transmission system and a method for encoding an input signal in the system are disclosed. According to the present invention, the VSB transmission system includes a convolutional encoder for encoding an input signal, a trellis-coded modulation (TCM) encoder for encoding the convolutionally encoded signal, and a signal mapper mapping the trellis-coded signal to generate a corresponding output signal. Different types of the convolutional encoders are explored, and the experimental results showing the performances of the VSB systems incorporating each type of encoders reveals that a reliable data transmission can be achieved even at a lower input signal to noise ratio when a convolutional encoder is used as an error-correcting encoder in a VSB system.