公开(公告)号：US20250110985A1

公开(公告)日：2025-04-03

申请号：US18478998

申请日：2023-09-30

Applicant: Microsoft Technology Licensing, LLC

Inventor： Justin James WAGLE ,  Rogerio BONATTI

IPC: G06F16/583 ,  G06F40/30 ,  G06V10/82

Abstract: Large language models (LLMs) are able to provide robust results based on specified formatting and organization. Traditionally, however, users must form detailed queries to obtain desired results in a desired format. Accordingly, although LLMs are designed to receive natural language input, users often lack the skill, knowledge, or patience to utilize LLMs to their full potential. Ambient information and user history associated with device screenshots are leveraged to provide proactive artificial-intelligence (AI) assistance and query resolution in an LLM environment. In particular, screenshots associated with a computer display are continuously captured and analyzed to detect activity triggers for plugins, for example. In response to detecting an activity trigger, local context associated with one or more prior screenshots is collected. The collected context is then used to inform the plugin for performing the task, thereby reducing the burden placed on the user to input the required information.