Systems and methods are provided for performing operations on an augmented reality (AR) device using an external vision system. The system establishes, by the AR device, a communication with an external client device. The system overlays, by the AR device, a first AR object on a real-world environment being viewed using the AR device. The system receives interaction data from the external client device representing movement of a user determined by the external client device. The system, in response to receiving the interaction data from the external client device, modifies the first AR object by the AR device.