The invention discloses a real-time speech translation system based on speech recognition. The real-time speech translation system comprises a speech acquisition module, a text conversion module, a sign language conversion module and a displaying module, wherein the speech acquisition module is used for acquiring speech information to be translated; the text conversion module is used for converting the speech information to be translated into target text information; the sign language conversion module is used for converting the target text information into target sign language information; and the displaying module is used for displaying the target text information and/or displaying the target sign language information. Therefore, when a hearing impaired person communicates with a normalhearing person, the speech information of the normal hearing person is acquired and converted into text information and sign language information, and the text information and the sign language information are displayed to the hearing impaired person by means of an intelligent mobile terminal, so that the hearing impaired person can understand the content of the speech of the normal hearing person, thereby realizing convenient, rapid and effective speech translation.