One embodiment of the present invention sets forth a technique for controlling the operations of a consumer electronic (CE) device. A system includes a display device, CE device, and a remote controller with a touchpad. The CE device is configured to output a context-specific user interface for display by the display device. The particular user interface varies based on the operational context of the CE device. The remote controller sends signals that are detected by the CE device and control operations performed by the CE device. More specifically, the remote controller is used as a handheld pointing device to select operations through interactions with the context-specific user interface. A user may apply gestures to the touchpad that are interpreted based on the context-specific user interface to control the CE device.