The invention discloses a detection method of a communication system with the space-time encoding in vertical division, which detects the communication system with the space-time encoding in vertical division by using an improved optimization algorithm of particle clusters, computes a fitness function of the optimization algorithm of particle clusters by using an ML detection formula that F(s) equals II r minus Hs II <2>, while s is (s1, s2,..., sM) and belongs to the Omega space, and achieves the goal of acquiring the detection performance which is close to that of the ML method with low complexity. The implementation of the detection method comprises the following steps: first step, the signal received by a baseband and a channel estimation value are input into a detector, algorithm parameters are set, and the speed and the position of the particles are initialized; second step, according to the detection formula, the fitness of the particles is computed, and an individual extreme value and a global extreme value are initialized; third step, the speed and the position of the particles are updated; fourth step, a variation operation is implemented on a position vector; fifth step, the individual extreme value and the global extreme value are updated; sixth step, the method determines whether a largest iteration number is reached, if yes, the method outputs the global extreme value which is the final detection value; if not, the method turns to the third step. The algorithm used by the invention has simplicity, low computation complexity, quicker computation speed and higher computation precision.