An “Interactive Virtual Display,” as described herein, provides various systems and techniques that facilitate ubiquitous user interaction with both local and remote heterogeneous computing devices. More specifically, the Interactive Virtual Display uses various combinations of small-size programmable hardware and portable or wearable sensors to enable any display surface (e.g., computer display devices, televisions, projected images/video from projection devices, etc.) to act as a thin client for users to interact with a plurality heterogeneous computing devices regardless of where those devices are located relative to the user. The Interactive Virtual Display provides a flexible system architecture that enables communication and collaboration between a plurality of both local and remote heterogeneous computing devices. This communication and collaboration enables a variety of techniques, such as adaptive screen compression, user interface virtualization, real-time gesture detection to improve system performance and overall user experience, etc.