Aspects extend to methods, systems, and computer program products for privilege-based task processing at a virtual assistant. An owner of a virtual assistant can assign other users privileges for interacting with the owner's calendar through the virtual assistant. To establish user privileges, the owner can enter information about himself or herself, including user permissions, into a Privileged Associates Database (“PAD”). An administrator can also configure some or all of this privileged associate(s) information to be entered automatically in the PAD. As such, a privileged associate can request that the virtual assistant schedule an event on the owner's calendar. In response, the virtual assistant refers to the PAD and checks privileges. The virtual assistant automatically synthesizes a response accordingly (e.g., scheduling the event, displaying schedule information, requesting owner confirmation, preventing the request, etc.) based on the owner's schedule and the associate's privileges.