The invention discloses an object tracking method based on a deep learning characteristic and a point-to-set distance metric learning. The object tracking method comprises the following steps of randomly selecting a plurality of objective samples and background samples in a tracked starting frame; performing object sample characteristic extraction on the object samples, and performing background sample characteristic extraction on the background samples; clustering the extracted objective sample characteristics for forming a plurality of objective template sets, clustering the extracted background sample characteristics for forming a plurality of background template sets; learning a projection matrix through reducing the distance between the samples of the same kind and increasing the distance between the samples of different kinds; performing objective candidate acquisition on the subsequent frame according to Gaussian distribution; extracting the characteristic of the objective candidate, and projecting the objective template sets, the background template sets and the objective candidate to a common subspace by means of the projection matrix; calculating the distance of each objective candidate to all objective template sets, wherein summation of the distances is used as the score of each objective candidate, and the final tracking result is the average value of a plurality of first objective candidates with least distances.