A method and an apparatus for detecting an abnormality of a caller are provided. The method includes at the beginning of a call, acquiring, by a terminal device, real voice/video data of a call object who needs abnormality detection and a corresponding pre-trained multi-stage neural network detection model, during the call, collecting, by the terminal device, call data according to a preset data collection policy, for each call object, inputting the currently collected call data and the real voice/video data of the call object into the model of the call object, and determining whether the call object is abnormal according to a detection result output by the model, in which the call data includes image data and/or voice data, and an identification manner adopted by the model includes face identification, voiceprint identification, limb movement identification, and/or lip language identification. By adopting the disclosure, the abnormality of the caller may be accurately detected, and the voice forgery and the video forgery mimicked by AI during a call may be accurately identified.