In one embodiment, a device classification service forms a device cluster by applying clustering to telemetry data associated with a plurality of devices. The service obtains device type labels for the device cluster. The service generates a device type classification rule using the device type labels and the telemetry data. The service determines whether the device type classification rule should be revalidated by applying a revalidation policy to the device type classification rule. The service revalidates the device type classification rule, based on a determination that the device type classification rule should be revalidated.