Systems and methods are provided for predicting system or network failures, such as a degradation in the services provided by a service provider system or network, a system or network outage, etc. In a discovery phase, failure scenarios that can be accurately predicted based on monitored system events are identified. In an operationalization phase, those failure scenarios can be used to design production run-time machines that can be used to predict, in real-time, a future failure scenario. An early warning signal(s) can be sent in advance of the failure scenario occurring.