Examples provide a video security system including an electronic processor configured to detect an anomaly by obtaining video data captured using an image sensor and determining a number of people detected in each of a plurality of frames of the video data. For each moment t, the electronic processor determines an average number of people n over a time interval (t−Δt1, t), where Δt1 is greater than or equal to the moment t. For each non-overlapping time interval Δt1, the electronic processor updates a histogram based on the average number of people for the time interval Δt1. In response to determining, using the histogram, that the probability ρ of the average number of people n over the time interval (t−Δt1, t) is less than a threshold probability ρ*, the electronic processor generates an alert indicating that an anomalous crowd size is detected.