Provided is a system and method of providing a sign language broadcasting service to viewers having hearing impairments, and more particularly, the system and method of providing a sign language broadcasting service that may enable a user to select an option of viewing only a broadcast image, or an option to viewing both the broadcast image and a sign language interpreting image, by transmitting the broadcast image and the sign language interpreting image using different transport streams.