A language model is constructed for mixed language expressions that have words from more than one natural language. Word equivalence probabilities for pairs of words among the languages are generated and stored. Word equivalence probabilities are used as required to generate a monolingual word history. The monolingual history is used by a monolingual language model to generate a next-word hypothesis. The word equivalence probabilities are also used to compute the next word probabilities in the foreign language.