Using a conversation context to manage conference participants including receiving a request from an invitee to join a conference in progress; extracting audio from the conference; converting said audio to data describing the conference; matching data to a context, wherein the context comprises a specification of a subject matter of the conference and a list of authorized participants for the conference; comparing the invitee to the list of the authorized participants for the conference; responsive to determining that the invitee matches an authorized participant of the list of authorized participants for the conference, granting permission to the invitee to join the conference; and responsive to determining that the invitee does not match an authorized participant of the list of authorized participants for the conference, denying the invitee access to the conference.