A surveillance system improves accuracy and robustness of abnormal behavior detection of a monitored object traversing a space includes a metadata processing module, a model building module, and a behavior assessment module. The metadata processing module generates trajectory information for a monitor object and determines attributes of the monitored object. The model building module at least one of generates and updates normal motion models based on at least one of the trajectory information, the attributes, and an abnormal behavior score. The behavior assessment module generates the abnormal behavior score based on one of a plurality of methods. A first one of the plurality of methods defines wrong direction behavior. A second one of the plurality of methods defines wandering/loitering behavior. A third one of the plurality of methods defines speeding behavior.