This disclosure describes a system for disambiguating between multiple potential users that may have performed an item action (e.g., item removal or item placement) at an inventory location. For example, if there are three picking agents (users) standing near an inventory location and one of the agents removes an item (item action) from the inventory location, the example systems and processes described herein may utilize various inputs to disambiguate between the users and determine which of the potential users performed the item action.