The invention provides a method and system for generating video description sentences. The method includes the steps of processing a video flow acquired through a convolution neural network, to obtain a feature vector corresponding to each frame of the video flow; calculating the correlation of the adjacent frames according to the feature vectors, and on this basis, conducting clustering processing for all the frames of the video flow, so as to dividing all the frames corresponding to the video flow into a plurality of categories; and inputting clustering results into a recurrent neural network to obtain description sentences corresponding to the video flow. The invention creatively adds the clustering process, can better describe local information of time granularities, and prevents the problem of the inevitable loss of the partial local information using equal interval sampling, that is the processing speed is increased under the premise of reducing the loss of the local information. Therefore, the invention effectively overcomes the drawbacks of the prior art and has a high industrial utilization value.