A technique for searching for probable matches in a video surveillance system is disclosed. A new event, such as a face captured in an image set, is matched against other events in a database of events. A similarity score is generated based on the difference between the new event and other events in the database. The similarity score may be weighted by information external to the image sets. Because of limited system resources, an association between a new event and every other event in the system may not be kept. Thus, when searching for probable matches of a particular event, some events that are related to the particular event may not be initially selected. Such events may be associated with an event in a first set of events that are associated with the particular event. Therefore, a second set of events is selected that are associated with the first set of events.