Embodiments are directed to monitoring network traffic in a network. A network monitoring engine may be employed to monitor the network to provide metric profiles based on a plurality of characteristics associated with one or more network flows. The network monitoring engine may provide profile objects based on the metric profiles. The network monitoring engine may provide the profile objects to a classifier engine. The classifier engine provide trained activity models selected from a plurality of trained activity models that may be based on a ranked ordering of characteristics of the trained activity models and the profile objects. The classifier engine may provide classification results for the profile objects based on the trained activity models. And, the network monitoring engine may execute policies based on the classification results associated with the profile objects.