Aspects of the disclosure allocate shares of processing resources or other physical resources among virtual machines (VMs) operating as, for example, virtual desktops on a plurality of host computing devices. Allocations of resources are adjusted based on the user activity, VM activity, and/or application activity detected by an agent executing on each VM. Allocated shares may be boosted, unboosted, or normalized, depending on the type and duration of detected activity, by a resource allocation manager executing on a management server.