The invention discloses a maximum likelihood space-time code mode blind identification method based on optimization of a penalty function. The maximum likelihood space-time code mode blind identification method based on optimization of the penalty function comprises the specific steps of (1) extracting a space-time code set, (2) obtaining a code parameter set, (3) calculating a parallel matrix, (4) building grouped correlation matrixes according to the parallel matrix, (5) obtaining characteristic value vectors according to the grouped correlation matrixes, (6) calculating penalty function values of code parameter combinations, obtaining likelihood function values, (7) repeatedly obtaining likelihood function value vectors corresponding to the code parameter combinations, and (8) selecting a code pattern with the minimum likelihood function value vector as a judgment code pattern. Due to the fact that a covariance matrix of a received signal is solved, characteristic values and characteristic vectors of the covariance matrix are obtained; a likelihood function is built according to space-time code parameters; penalty function values of possible code patterns are calculated to obtain the likelihood function values, and a code pattern with the minimum likelihood function value is selected as an identification code pattern. Due to the fact that correction is carried out on the likelihood function to achieve the purpose of improving calculation accuracy, the blind identification accuracy of an encoding mode is improved, and stability of a system is improved.