A media device manages transitions between executing an application in augmented reality (AR) and virtual reality (VR) modes. In the AR mode, an AR view of a scene is rendered that includes real world objects of the scene and optionally includes one or more virtual objects. A user may interact with the virtual objects using a pointing controller or other input device. When transitioning to a VR mode, the media device determines one or more real world objects that can be replaced by virtual replacement objects in a VR view of the scene, and determines one or more real world objects that can be removed from the VR view of the scene. A state of the virtual objected when exiting the AR mode may be stored, and the virtual objects may be rendered in the VR mode based on their stored state.