The embodiment of the invention discloses a multimedia data processing method and device. The method comprises the steps of acquiring multiple audio file samples corresponding to multimedia data, and generating target audio energy matrixes corresponding to the audio file samples respectively, wherein the audio file samples have the same frame number, and elements in each target audio energy matrix are audio energy values of different frames in a corresponding target audio file sample respectively; conducting average value processing on the audio energy values of different frames in all the target audio energy matrixes, so as to obtain a target average value matrix, wherein elements in the target average value matrix are audio energy average values of different frames; and determining a frame corresponding to the maximum audio energy average value in the target average value matrix as a target frame, and determining the refrain starting moment of the multimedia data according to the target frame. In this way, accurate position of the refrain of the multimedia data can be achieved through multiple audio file samples.