The invention provides a video face recognition method based on attention set metric learning (ASML), which mainly comprises steps of ASML, memory attention weight and natural integration of the ASML to a convolutional neural network (CNN). The process is that an effective distance metric on an image set is defined firstly, the distance within a set is minimized significantly, the concentration distance is maximized at the same time, the weight is used as a nerve Turing machine, a face feature set is used as a memory, the weight is used as an address read-write memory, the ASML is finally naturally integrated to the convolutional neural network, and an end-to-end learning scheme is thus formed. The gap between probability distributions of the same set is narrowed, and the gap between different sets is expanded at the same time; sample offset and noise in the video or image set is reduced, the information in the video or image set is used effectively, and the recognition performance is thus improved.