The invention provides an image natural language description generation method and device with cross-linguistic learning ability. The method comprises the steps that English description sentences are translated into target language description sentences through a machine; part of the target language description sentences are selected randomly to form a training sample set; a smooth sample set and a non-smooth sample set are used for training a sentence smooth degree model; the target language description sentences in a candidate data set are subjected to smooth degree evaluation through the sentence smooth degree model, and according to the smooth degree probability of each target language description sentence, a strategy for training an image description sentence generation model is set; according to the strategy, the image description sentence generation model is trained, and the trained image description sentence generation model is obtained. According to the image natural language description generation method and device with the cross-linguistic learning ability, the aim is achieved that the image description sentence generation model of a target language is guided, trained and generated dependent on smooth degree evaluation results, the influence of non-smooth target language description sentences on the training process is lowered, and the accuracy of the image description sentence generation model of the target language is improved.