This disclosure describes a performance-monitoring system that computes a display performance metric of a remote application. During operation, the system performs a sequence of input events, and receives information which updates a graphical user interface (GUI). The GUI displays a sequence of frames rendered by a remote application in response to the input events. The system then samples colors at a number of pivot points on the GUI, and matches the a respective frame to a previously performed input event based on the sampled colors. The system subsequently computes a display performance metric for the remote application based on the frames and the corresponding input events.