An adaptive biometric authentication system may include a user identity reference module that is configured to maintain user identification items stored in a secure memory. The system may include a passive data aggregator that is configured to receive and aggregate data items that are passively collected by a device that is in proximity to a user. The system may include a user identity confidence level generator that is configured to generate a user identity confidence level that indicates a confidence that the user in proximity to the device is the reference user based at least in part on a comparison between the passively collected data items and the user identification items, and to update the user identity confidence level as additional passively collected data items are received. The system may include an authentication module that facilitates user authentication based at least in part on the user identity confidence level.