An online system predicts a number of interruption events within a time period and identifies anomalous numbers of interruption events using an interruption prediction model. The online concierge system maintains application state data that describes a state of an application workflow for a client application. The online concierge system identifies interruption events that represent interruptions to the application workflow and logs interruption events in an interruption log, wherein each entry of the interruption log describes an interruption event and its corresponding state. The online concierge system predicts a number of interruption events that will occur within a time period based on an interruption prediction model. The online concierge system computes an actual number of interruption events that occurred during the time period and computes a difference between the actual number and the predicted number. If the difference exceeds a threshold value, the online concierge system performs a remedial action.