Some embodiments of the invention provide a method that performs security operations for packets that are processed by a forwarding element. The method of some embodiments receives, at a security agent operating on a physical machine, a packet from a forwarding element that also operates on the physical machine. The method then determines whether a security rule is stored for the packet at the security agent. When no security rule is stored for the packet, the method transmits the packet to a default security controller of several security controllers that store security rules for a network and process packets according to the stored security rules. When the security rule is stored for the packet, the method processes the packet according to the stored security rule for the packet.