A method, system, and computer program product for autonomously controlling caller access to a teleconference based on specified preferences of a moderator. Using preferences established by the moderator of a teleconference, a conference control system may grant or restrict access of participants of the teleconference. The conference control system may authorize callers of a teleconference based on the current agenda item being discussed or the amount of time a caller has been on the ongoing teleconference. When a specific caller is not authorized to be on a conversation, the conference control system may mute the conversation for a specific caller, or disconnect the caller. The conference control system may autonomously re-connect the caller when an authorized agenda item for the caller is being discussed.