The invention discloses a physical layer security algorithm of an OFDM (orthogonal frequency division multiplexing) system based on double matrix transformation. The physical layer security algorithm comprises a physical layer encryption algorithm and a physical layer decryption algorithm. The physical layer security algorithm includes the steps: firstly, taking initial values of a chaotic sequence as seed keys to generate two complex diagonal key matrixes; secondly, multiplying one key diagonal matrix for a symbolic vector mapped by a constellation; finally, performing IFFT (inverse fast Fourier transform) on the symbolic vector, and multiplying another different or same key diagonal matrix for an output vector subjected to the IFFT to realize physical layer encryption. Theoretical analysis and simulation experiment results show that the physical layer security algorithm can resist plaintext-ciphertext attack and attack in other modes and is simple as compared with a traditional link layer security algorithm, practical requirements of next generation wireless communication for broadband and amalgamation can be met, security of data, air interfaces and wireless links can be effectively protected, and the security algorithm hardly has any effect on original performances of the system.