Eyewear providing an interactive augmented reality experience between two users of eyewear devices to perform a shared group task. During a shared group task session, each eyewear user can manipulate virtual objects displayed in a respective virtual scene that is viewable to each user to perform collaboration. The virtual objects can include many different types of objects, such as a building structure that can be jointly created and edited by the eyewear users.