A conversation system may dynamically control a conversation or workflow by performing multi-modal conversation monitoring, generating actions that control the conversation based on the multi-modal monitoring producing conversation elements that deviate from patterns of a selected plan for that conversation, and/or by dynamically generating and/or updating the plan for future conversations based on the pattern recognition. For instance, the conversation system may detect a pattern within completed conversations that resulted in a common outcome, may monitor an active conversation between at least an agent and a participant, may extract different sets of conversation elements from different points in the active conversation, may determine that a particular set of conversation elements deviates from the pattern, and may modify the active conversation by performing one or more actions based on the particular set of conversation elements that deviate from the pattern.