The invention provides a monitoring system for judging a video behavior based on deep learning. The system comprises a monitoring terminal and a server; the monitoring terminal comprises a camera anda front end alarm module; the server comprises a cloud platform video database, an offline video database, a deep learning judgment module and an abnormal event alarm module; the camera is used for collecting video information in real time; the cloud platform video database stores the video information sent by the camera; the offline video database stores the video information for the deep learning judgment module to construct a behavior judgment model; the deep learning judgment module constructs the behavior judgment model according to the videos in the offline video database and judges thereceived video behavior according to the behavior judgment model; the front end alarm module is an alarm light or a horn; the abnormal event alarm mode is an alarm light or a horn; and the deep learning judgment module comprises a feature extraction module, a feature fusing and splicing module, a semantic coding determining modules, a feature decoding modules and a classification alarm module.