An ultra-low latency remote access system may stream video images from a remote application server to a client device, but may render a cursor separately on the client device. The remote application may send cursor bitmaps and other cursor information to the client device, which may be rendered by the client device to give a near-native response time and feel to the user experience. The remote access system may operate in a browser environment, with executable code in the browser to capture user input, including pointer and keyboard inputs. Executable code on the application server may encode screen images into video, which may be transmitted and rendered on the client device. In some cases, the rendered video may be larger than the displayable area of the client device, allowing for panning and zooming to be performed on the client device.