Provided is a speech signal classification system and method. The speech signal classification system includes a primary recognition unit for determining using characteristics extracted from a speech frame whether the speech frame is a voice sound, a non-voice sound, or background noise and a secondary recognition unit for determining using at least one other speech frame whether a determination-reserved speech frame is an non-voice sound or background noise, if it is determined according to a primary recognition result that an input speech frame is not a voice sound. The system reserves a determination of the input speech frame, stores characteristics of at least one other speech frame to determine the determination-reserved speech frame, calculates secondary statistical values from characteristics of the determination-reserved speech frame and the stored characteristics of the other speech frames, and determines using the calculated secondary statistical values whether the determination-reserved speech frame is an non-voice sound or background noise. Accordingly, if an input speech frame is not a voice sound, the input speech frame can be more accurately classified and output as an non-voice sound or background noise, and thus errors, which may be generated in determination of a signal corresponding to an non-voice sound, can be reduced.