A system for detecting an anomaly in a video of a factory automation scene is disclosed. The system may accept the video; accept a set of training feature vectors derived from spatio-temporal regions of a training video, where a spatio-temporal region is associated with one or multiple training feature vectors; partition the video into multiple sequences of video volumes; produce a sequence of binary difference images for each of the video volumes; count occurrences of each of predetermined patterns of pixels in each binary difference image for each of the video volumes to produce an input feature vector including an input motion feature vector defining a temporal variation of counts of the predetermined patterns for each of the video volumes; produce a set of distances based on the produced input feature vectors and the set of training feature vectors; and detect the anomaly based on the produced set of distances.