Embodiments of the disclosure provide for systems and methods for searching video data for events and/or behaviors. An inference engine can be used to aide in the searching. In some embodiments, a user can specify various search criteria, for example, a video source(s), an event(s) or behavior(s) to search, and an action(s) to perform in the event of a successful search. The search can be performed by analyzing an object(s) found within scenes of the video data. An object can be identified by a number of attributes specified by the user. Once the search criteria has been received from the user, the video data can be received (or extracted from storage), the data analyzed for the specified events (or behaviors), and the specified action performed in the event a successful search occurs.