A method for adapting a guest mode for portable speakers includes obtaining a current state of an assistant-enabled device while a guest mode is enabled. Here, the assistant-enabled device is configured to allow a digital assistant to fulfill queries spoken by speakers who are not enrolled users when the guest mode is enabled. Based on the current state of the assistant-enabled device, the method includes disabling the guest mode to suppress fulfillment of queries spoken by speakers who are not enrolled users. While the guest mode is disabled, the method includes receiving audio data corresponding to a query, the query including a command for the digital assistant to perform an action, performing speaker identification to determine that a speaker of the query is not an enrolled user, and based on determining that the speaker of the query is not an enrolled user, suppressing fulfillment of the action.