Low latency hand-tracking in augmented reality systems

    公开(公告)号:US12002168B2

    公开(公告)日:2024-06-04

    申请号:US17844541

    申请日:2022-06-20

    Applicant: Snap Inc.

    Abstract: A method for reducing motion-to-photon latency for hand tracking is described. In one aspect, a method includes accessing a first frame from a camera of an Augmented Reality (AR) device, tracking a first image of a hand in the first frame, rendering virtual content based on the tracking of the first image of the hand in the first frame, accessing a second frame from the camera before the rendering of the virtual content is completed, the second frame immediately following the first frame, tracking, using the computer vision engine of the AR device, a second image of the hand in the second frame, generating an annotation based on tracking the second image of the hand in the second frame, forming an annotated virtual content based on the annotation and the virtual content, and displaying the annotated virtual content in a display of the AR device.

    EXTERNAL SCREEN STREAMING FOR AN EYEWEAR DEVICE

    公开(公告)号:US20240119679A1

    公开(公告)日:2024-04-11

    申请号:US17960627

    申请日:2022-10-05

    Applicant: Snap Inc.

    CPC classification number: G06T19/006 G06T2200/24 G06T2219/024

    Abstract: Systems and methods are provided for performing operations on an augmented reality (AR) device using an external screen streaming system. The system establishes, by one or more processors of an AR device, a communication with an external client device. The system causes overlay of, by the AR device, a first AR object on a real-world environment being viewed using the AR device. The system receives, by the AR device, a first image from the external client device. The system, in response to receiving the first image from the external client device, overlays the first image on the first AR object by the AR device.

    DYNAMIC OVER-RENDERING IN LATE-WARPING

    公开(公告)号:US20220375028A1

    公开(公告)日:2022-11-24

    申请号:US17530242

    申请日:2021-11-18

    Applicant: Snap Inc.

    Abstract: A method for adjusting an over-rendered area of a display in an AR device is described. The method includes identifying an angular velocity of a display device, a most recent pose of the display device, previous warp poses, and previous over-rendered areas, and adjusting a size of a dynamic over-rendered area based on a combination of the angular velocity, the most recent pose, the previous warp poses, and the previous over-rendered areas.

    LATE WARPING TO MINIMIZE LATENCY OF MOVING OBJECTS

    公开(公告)号:US20220375026A1

    公开(公告)日:2022-11-24

    申请号:US17518828

    申请日:2021-11-04

    Applicant: Snap Inc.

    Abstract: A method for minimizing latency of moving objects in an augmented reality (AR) display device is described. In one aspect, the method includes determining an initial pose of a visual tracking device, identifying an initial location of an object in an image that is generated by an optical sensor of the visual tracking device, the image corresponding to the initial pose of the visual tracking device. rendering virtual content based on the initial pose and the initial location of the object, retrieving an updated pose of the visual tracking device, tracking an updated location of the object in an updated image that corresponds to the updated pose, and applying a time warp transformation to the rendered virtual content based on the updated pose and the updated location of the object to generate transformed virtual content.

    Remote annotation and navigation using an AR wearable device

    公开(公告)号:US12266060B2

    公开(公告)日:2025-04-01

    申请号:US18046367

    申请日:2022-10-13

    Applicant: Snap Inc.

    Abstract: Systems, methods, and computer readable media for remote annotations, drawings, and navigation instructions sent to an augmented reality (AR) wearable device from a computing device are disclosed. The AR wearable device captures images and sends them to the remote computing device to provide a real-time view of what the user of the AR wearable device sees. A user of the remote computing device can add navigation instructions and can select an image to annotate or draw on. The AR wearable device provides 3-dimensional (3D) coordinate information within a 3D world of the AR wearable device for the selected image. The user of the remote computing device then annotates or draws on the selected image. The remote computing device determines 3D coordinates for the annotations and drawings within the 3D world of the AR wearable device. The annotations and drawings are sent to the AR wearable device with associated 3D coordinates.

    Late warping to minimize latency of moving objects

    公开(公告)号:US12067693B2

    公开(公告)日:2024-08-20

    申请号:US17518828

    申请日:2021-11-04

    Applicant: Snap Inc.

    CPC classification number: G06T3/18 G06T13/40 G06T13/80 G06T15/205 G06T2210/44

    Abstract: A method for minimizing latency of moving objects in an augmented reality (AR) display device is described. In one aspect, the method includes determining an initial pose of a visual tracking device, identifying an initial location of an object in an image that is generated by an optical sensor of the visual tracking device, the image corresponding to the initial pose of the visual tracking device, rendering virtual content based on the initial pose and the initial location of the object, retrieving an updated pose of the visual tracking device, tracking an updated location of the object in an updated image that corresponds to the updated pose, and applying a time warp transformation to the rendered virtual content based on the updated pose and the updated location of the object to generate transformed virtual content.

    LOW LATENCY HAND-TRACKING IN AUGMENTED REALITY SYSTEMS

    公开(公告)号:US20240273843A1

    公开(公告)日:2024-08-15

    申请号:US18645277

    申请日:2024-04-24

    Applicant: Snap Inc.

    Abstract: A method for reducing motion-to-photon latency for hand tracking is described. In one aspect, a method includes accessing a first frame from a camera of an Augmented Reality (AR) device, tracking a first image of a hand in the first frame, rendering virtual content based on the tracking of the first image of the hand in the first frame, accessing a second frame from the camera before the rendering of the virtual content is completed, the second frame immediately following the first frame, tracking, using the computer vision engine of the AR device, a second image of the hand in the second frame, generating an annotation based on tracking the second image of the hand in the second frame, forming an annotated virtual content based on the annotation and the virtual content, and displaying the annotated virtual content in a display of the AR device.

    AUGMENTED REALITY SHARED SCREEN SPACE
    19.
    发明公开

    公开(公告)号:US20240193875A1

    公开(公告)日:2024-06-13

    申请号:US18078522

    申请日:2022-12-09

    Applicant: Snap Inc.

    CPC classification number: G06T19/006 H04L67/131

    Abstract: Systems, methods, and computer readable media for an augmented reality (AR) shared screen space. Examples relate to a host augmented realty (AR) device sharing a screen and a relative location of the AR device to the screen with guest AR devices where the guest AR devices share a relative location of the guest AR devices to a copy of the screen displayed on the display of the guest AR devices and where the users of the AR devices may see each other's location with the use of avatars around the shared screen and add augmentations to the shared screen. The yaw, roll, and pitch of the head of the avatars tracks the movement of the head of the user of the AR wearable device.

    Dynamic over-rendering in late-warping

    公开(公告)号:US11880958B2

    公开(公告)日:2024-01-23

    申请号:US18116665

    申请日:2023-03-02

    Applicant: Snap Inc.

    CPC classification number: G06T3/40 G06T19/006

    Abstract: A method for adjusting an over-rendered area of a display in an AR device is described. The method includes identifying an angular velocity of a display device, a most recent pose of the display device, previous warp poses, and previous over-rendered areas, and adjusting a size of a dynamic over-rendered area based on a combination of the angular velocity, the most recent pose, the previous warp poses, and the previous over-rendered areas.

Patent Agency Ranking