A system authenticates users using voice-based conversations. The system allows the authentication process to be customized using an authentication plan. For example, the system may be a multi-tenant system that allows customization of the authentication process for each tenant. The authentication plan is represented as an expression of phrase types, each phrase type associated with a phrase verification method. The system authenticates a user by executing the expression of an authentication plan for that user in response to a request from the user. The system performs a conversation with the user according to the authentication plan. The system determines whether to allow or deny the user request based on the result of evaluation of the expression of the authentication plan.