公开(公告)号：US20230393657A1

公开(公告)日：2023-12-07

申请号：US18326854

申请日：2023-05-31

Applicant: GOOGLE LLC

Inventor： Ruofei Du ,  Maksym Dzitsiuk

IPC: G06F3/01 ,  G06V20/50 ,  G06V10/77 ,  G06T7/70

CPC classification number: G06F3/013 ,  G06V20/50 ,  G06V10/7715 ,  G06T7/70

Abstract: A method including determining a gaze direction of a user of a wearable device, capturing an image using a forward-looking camera of the wearable device, detecting a surroundings of the user based on the image, determining whether or not the user is distracted based on the gaze direction and the surroundings, and in response to determining the user is distracted, causing an operation to be performed on the wearable device, the operation configured to cause the user to change the user's attention.

公开(公告)号：US20230305672A1

公开(公告)日：2023-09-28

申请号：US17656818

申请日：2022-03-28

Applicant: Google LLC

Inventor： Ruofei Du ,  Alex Olwal ,  Mathieu Simon Le Goc ,  David Kim ,  Danhang Tang

IPC: G06F3/04815 ,  G10L15/22 ,  G02B27/01

CPC classification number: G06F3/04815 ,  G10L15/22 ,  G02B27/0172 ,  G02B27/0176 ,  G02B2027/0178 ,  G10L2015/223

Abstract: Systems and methods are provided in which physical objects in the ambient environment can function as user interface implements in an augmented reality environment. A physical object detected within a field of view of a camera of a computing device may be designated as a user interface implement in response to a user command. User interfaces may be attached to the designated physical object, to provide a tangible user interface implement for user interaction with the augmented reality environment.

公开(公告)号：US20220414834A1

公开(公告)日：2022-12-29

申请号：US17756700

申请日：2020-05-26

Applicant: Google LLC

Inventor： Ruofei Du ,  David Kim

IPC: G06T5/00 ,  G06T7/50 ,  G06T7/13

Abstract: A method including receiving an image as a portion of a real-world space, placing an anchor on the image, determining a position of the anchor, determine a depth associated with the position of the anchor, applying an image editing algorithm based on the depth to the captured image, and rendering the edited image.

公开(公告)号：US20240402803A1

公开(公告)日：2024-12-05

申请号：US18800844

申请日：2024-08-12

Applicant: GOOGLE LLC

Inventor： Ruofei Du ,  Maksym Dzitsiuk

IPC: G06F3/01 ,  G06T7/70 ,  G06V10/77 ,  G06V20/50

Abstract: A method including determining a gaze direction of a user of a wearable device, capturing an image using a forward-looking camera of the wearable device, detecting a surroundings of the user based on the image, determining whether or not the user is distracted based on the gaze direction and the surroundings, and in response to determining the user is distracted, causing an operation to be performed on the wearable device, the operation configured to cause the user to change the user's attention.

公开(公告)号：US11967335B2

公开(公告)日：2024-04-23

申请号：US17446877

申请日：2021-09-03

Applicant: Google LLC

Inventor： Ruofei Du ,  Hendrik Wagenaar ,  Alex Olwal

IPC: G06F3/01 ,  G06T7/50 ,  G06T7/70 ,  G06V40/16 ,  G06V40/19 ,  G10L15/22 ,  G10L15/25 ,  G10L15/26 ,  G10L21/10 ,  G10L21/18 ,  H04R1/08 ,  H04R1/40 ,  H04R3/00

CPC classification number: G10L21/10 ,  G06F3/013 ,  G06T7/50 ,  G06T7/70 ,  G06V40/171 ,  G06V40/19 ,  G10L15/22 ,  G10L15/25 ,  G10L15/26 ,  G10L21/18 ,  H04R1/08 ,  H04R1/406 ,  H04R3/005 ,  G06T2207/30201 ,  H04R2201/401

Abstract: An augmented reality (AR) device, such as AR glasses, may include a microphone array. The sensitivity of the microphone array can be directed to a target by beamforming, which includes combining the audio of each microphone of the array in a particular way based on a location of the target. The present disclosure describes systems and methods to determine the location of the target based on a gaze of a user and beamform the audio accordingly. This eye-tracked beamforming (i.e., foveated beamforming) can be used by AR applications to enhance sounds from a gaze direction and to suppress sounds from other directions. Additionally, the gaze information can be used to help visualize the results of an AR application, such as speech-to-text.

公开(公告)号：US20240129437A1

公开(公告)日：2024-04-18

申请号：US18047420

申请日：2022-10-18

Applicant: Google LLC

Inventor： Yinda Zhang ,  Ruofei Du

IPC: H04N7/15 ,  G06F3/16 ,  G06T13/00

CPC classification number: H04N7/157 ,  G06F3/167 ,  G06T13/00

Abstract: A method can include selecting, from at least a first avatar and a second avatar based on at least one attribute of a calendar event associated with a user, a session avatar, the first avatar being based on a first set of images of a user wearing a first outfit and the second avatar being based on a second set of images of the user wearing a second outfit, and presenting the session avatar during a videoconference, the presentation of the session avatar changing based on audio input received from the user during the videoconference.

公开(公告)号：US20240062046A1

公开(公告)日：2024-02-22

申请号：US18270685

申请日：2021-03-31

Applicant: Google LLC

Inventor： Ruofei Du ,  Yinda Zhang ,  Weihao Zeng

IPC: G06N3/0464 ,  G06V10/82 ,  G06V10/42 ,  G06V10/44 ,  G06N3/084

CPC classification number: G06N3/0464 ,  G06V10/82 ,  G06V10/42 ,  G06V10/44 ,  G06N3/084

Abstract: A system including a computer vision model configured to perform a machine learning task is described. The computer vision model includes multiple wrapped convolutional layers, in which each wrapped convolutional layer includes a respective convolutional layer configured to receive, for each time step of multiple time steps, a layer input and to process the layer input to generate an initial output for the current time step, and a respective note-taking module configured to receive the initial output and to process the initial output to generate a feature vector for the current time step, the feature vector representing local information of the wrapped convolutional layer. The model includes a summarization module configured to receive the feature vectors and to process the feature vectors to generate a revision vector for the current time step, the revision vector representing global information of the plurality of wrapped convolutional layers.

公开(公告)号：US20240053877A1

公开(公告)日：2024-02-15

申请号：US17818852

申请日：2022-08-10

Applicant: Google LLC

Inventor： Ruofei Du ,  Na Li ,  Jing Jin ,  Maria Mandlis ,  Scott Joseph Miles ,  Ping Yu

IPC: G06F3/04847 ,  G06N3/04 ,  G06F3/0482 ,  G06F3/0486

CPC classification number: G06F3/04847 ,  G06N3/04 ,  G06F3/0482 ,  G06F3/0486

Abstract: A method includes providing an interactive graphical user interface comprising a first menu providing one or more input options, a second menu providing one or more machine learning models, and a third menu providing one or more output formats. The method also includes generating a graph in a portion of the interactive graphical user interface by detecting one or more user selections of an input option, a machine learning model, and an output format, displaying nodes corresponding to the input option, the machine learning model, the output format, and displaying edges connecting the first node to the second node, and the second node to the third node. The method additionally includes applying the machine learning model to an input associated with the input option to generate an output in the output format. The method further includes providing, by the interactive graphical user interface, the output in the output format.

公开(公告)号：US11868583B2

公开(公告)日：2024-01-09

申请号：US17656818

申请日：2022-03-28

Applicant: Google LLC

Inventor： Ruofei Du ,  Alex Olwal ,  Mathieu Simon Le Goc ,  David Kim ,  Danhang Tang

IPC: G06F3/04815 ,  G02B27/01 ,  G10L15/22

CPC classification number: G06F3/04815 ,  G02B27/0172 ,  G02B27/0176 ,  G10L15/22 ,  G02B2027/0178 ,  G10L2015/223

Abstract: Systems and methods are provided in which physical objects in the ambient environment can function as user interface implements in an augmented reality environment. A physical object detected within a field of view of a camera of a computing device may be designated as a user interface implement in response to a user command. User interfaces may be attached to the designated physical object, to provide a tangible user interface implement for user interaction with the augmented reality environment.

公开(公告)号：US20230206568A1

公开(公告)日：2023-06-29

申请号：US18000237

申请日：2020-06-22

Applicant: GOOGLE LLC

Inventor： Ruofei Du ,  David Kim

IPC: G06T19/00 ,  G06T7/50 ,  G06T15/50 ,  G06T7/90 ,  G06T7/70 ,  G06T19/20

CPC classification number: G06T19/006 ,  G06T7/50 ,  G06T15/506 ,  G06T7/90 ,  G06T7/70 ,  G06T19/20 ,  G06T2219/2012 ,  G06T2207/10028

Abstract: A system and method for relighting of an augmented reality scene are provided. Image data and depth data corresponding to a physical environment may be captured by an image sensor of a computing device, for display in an AR scene generated by the computing device. The system may use the image data and depth data to generate a live depth map of the physical environment. A virtual light source may be placed in the AR scene of the physical environment in response to a user input. The system may relight the AR scene in response to the placement of the virtual light source based on the image data and the depth map.