The invention relates to a flame detection method based on video time and spatial information, which belongs to the technical field of image information processing. First, a prospect motion image is separated from a source image, and then pixels with flame color characteristics are found out from the prospect motion image to form a suspected flame area; then observation points are arranged in the pixels on the edge of the suspected flame area with the flashing characteristics of the pixels on the edge of a real flame area, a hidden markov model is built, and time and space pattern characteristics are extracted to form an observation characteristic value sequence; the observation characteristic value sequence is input into a hidden markov mode decision device which is trained in advance to obtain the decision result whether the observation points are real flame pixels; and when the observation points which account quite a big part of all the observation points are decided to be the real flame pixels, the suspected flame area is thought to be the real flame area, and alarm is given. The flame detection method based on video time and spatial information has lower missing detection rate and error detection rate, very good robustness and can be used for more fire monitoring occasions.