公开(公告)号：US10733447B2

公开(公告)日：2020-08-04

申请号：US16515891

申请日：2019-07-18

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Tomasz Jan Malisiewicz ,  Daniel DeTone

IPC: G06K9/00 ,  A63F13/211 ,  A63F13/212 ,  A63F13/213 ,  A63F13/00 ,  G06F1/16 ,  G06F3/01 ,  G06F3/0338 ,  G06F3/0346 ,  G06F3/0484 ,  G06N3/00 ,  G06N3/04 ,  G06K9/62 ,  G06K9/46 ,  G06N3/08 ,  A63F13/428 ,  G02B27/01 ,  G06N5/00 ,  G06N7/00

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

公开(公告)号：US20190340435A1

公开(公告)日：2019-11-07

申请号：US16515891

申请日：2019-07-18

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Tomasz Jan Malisiewicz ,  Daniel DeTone

IPC: G06K9/00 ,  G06K9/62 ,  G06N3/04 ,  G06F3/0484 ,  G06F3/0346 ,  G06F3/0338 ,  G06F1/16 ,  A63F13/00 ,  G06N3/08 ,  G06K9/46 ,  G06F3/01 ,  G06N3/00 ,  A63F13/212 ,  A63F13/213 ,  A63F13/211

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

公开(公告)号：US20190147341A1

公开(公告)日：2019-05-16

申请号：US16190948

申请日：2018-11-14

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Daniel DeTone ,  Tomasz Jan Malisiewicz

IPC: G06N3/08 ,  G06N3/04 ,  G06T7/00

Abstract: Systems, devices, and methods for training a neural network and performing image interest point detection and description using the neural network. The neural network may include an interest point detector subnetwork and a descriptor subnetwork. An optical device may include at least one camera for capturing a first image and a second image. A first set of interest points and a first descriptor may be calculated using the neural network based on the first image, and a second set of interest points and a second descriptor may be calculated using the neural network based on the second image. A homography between the first image and the second image may be determined based on the first and second sets of interest points and the first and second descriptors. The optical device may adjust virtual image light being projected onto an eyepiece based on the homography.

公开(公告)号：US20240231102A1

公开(公告)日：2024-07-11

申请号：US18417523

申请日：2024-01-19

Applicant: Magic Leap, Inc.

Inventor： Daniel DeTone ,  Tomasz Jan MALISIEWICZ ,  Andrew Rabinovich

IPC: G02B27/01 ,  G06N3/08 ,  G06T7/33 ,  G06T7/73 ,  G06V10/40

CPC classification number: G02B27/0172 ,  G06N3/08 ,  G06T7/33 ,  G06T7/74 ,  G06V10/40 ,  G06T2207/10016 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/30244

Abstract: In an example method of training a neural network for performing visual odometry, the neural network receives a plurality of images of an environment, determines, for each image, a respective set of interest points and a respective descriptor, and determines a correspondence between the plurality of images. Determining the correspondence includes determining one or point correspondences between the sets of interest points, and determining a set of candidate interest points based on the one or more point correspondences, each candidate interest point indicating a respective feature in the environment in three-dimensional space). The neural network determines, for each candidate interest point, a respective stability metric and a respective stability metric. The neural network is modified based on the one or more candidate interest points.

公开(公告)号：US11893789B2

公开(公告)日：2024-02-06

申请号：US17288877

申请日：2019-11-14

Applicant: Magic Leap, Inc.

Inventor： Danying Hu ,  Daniel DeTone ,  Tomasz Jan Malisiewicz

IPC: G06V20/20 ,  G06T7/80 ,  G06T7/70 ,  G06N3/08 ,  G06T3/40 ,  G06T9/00 ,  G06N3/045 ,  G02B27/01 ,  G06T19/00 ,  G06V10/10 ,  G06V10/82 ,  G06V10/22 ,  G06V40/10

CPC classification number: G06V20/20 ,  G02B27/0172 ,  G06N3/045 ,  G06N3/08 ,  G06T3/4046 ,  G06T7/70 ,  G06T7/80 ,  G06T9/002 ,  G06T19/006 ,  G06V10/17 ,  G06V10/225 ,  G06V10/82 ,  G06V40/10 ,  G06T2207/30244

Abstract: A deep neural network provides real-time pose estimation by combining two custom deep neural networks, a location classifier and an ID classifier, with a pose estimation algorithm to achieve a 6D0F location of a fiducial marker. The locations may be further refined into subpixel coordinates using another deep neural network. The networks may be trained using a combination of auto-labeled videos of the target marker, synthetic subpixel corner data, and/or extreme data augmentation. The deep neural network provides improved pose estimations particularly in challenging low-light, high-motion, and/or high-blur scenarios.

公开(公告)号：US20220067378A1

公开(公告)日：2022-03-03

申请号：US17472151

申请日：2021-09-10

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Tomasz Jan Malisiewicz ,  Daniel DeTone

IPC: G06K9/00 ,  A63F13/211 ,  A63F13/212 ,  A63F13/213 ,  A63F13/00 ,  G06F1/16 ,  G06F3/01 ,  G06F3/0338 ,  G06F3/0346 ,  G06F3/0484 ,  G06N3/00 ,  G06N3/04 ,  G06K9/62 ,  G06K9/46 ,  G06N3/08

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

公开(公告)号：US11120266B2

公开(公告)日：2021-09-14

申请号：US16916554

申请日：2020-06-30

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Tomasz Jan Malisiewicz ,  Daniel DeTone

IPC: G06K9/00 ,  A63F13/211 ,  A63F13/212 ,  A63F13/213 ,  A63F13/00 ,  G06F1/16 ,  G06F3/01 ,  G06F3/0338 ,  G06F3/0346 ,  G06F3/0484 ,  G06N3/00 ,  G06N3/04 ,  G06K9/62 ,  G06K9/46 ,  G06N3/08 ,  A63F13/428 ,  G02B27/01 ,  G06N5/00 ,  G06N7/00

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

公开(公告)号：US10977554B2

公开(公告)日：2021-04-13

申请号：US16190948

申请日：2018-11-14

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Daniel DeTone ,  Tomasz Jan Malisiewicz

IPC: G06K9/00 ,  G06N3/08 ,  G06N3/04 ,  G06T7/00 ,  G06K9/46 ,  G06K9/62

Abstract: Systems, devices, and methods for training a neural network and performing image interest point detection and description using the neural network. The neural network may include an interest point detector subnetwork and a descriptor subnetwork. An optical device may include at least one camera for capturing a first image and a second image. A first set of interest points and a first descriptor may be calculated using the neural network based on the first image, and a second set of interest points and a second descriptor may be calculated using the neural network based on the second image. A homography between the first image and the second image may be determined based on the first and second sets of interest points and the first and second descriptors. The optical device may adjust virtual image light being projected onto an eyepiece based on the homography.

公开(公告)号：US20200334461A1

公开(公告)日：2020-10-22

申请号：US16916554

申请日：2020-06-30

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Tomasz Jan Malisiewicz ,  Daniel DeTone

IPC: G06K9/00 ,  A63F13/211 ,  A63F13/212 ,  A63F13/213 ,  A63F13/00 ,  G06F1/16 ,  G06F3/01 ,  G06F3/0338 ,  G06F3/0346 ,  G06F3/0484 ,  G06N3/00 ,  G06N3/04 ,  G06K9/62 ,  G06K9/46 ,  G06N3/08

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

公开(公告)号：US20200097819A1

公开(公告)日：2020-03-26

申请号：US16588505

申请日：2019-09-30

Applicant: Magic Leap, Inc.

Inventor： Daniel DeTone ,  Tomasz Malisiewicz ,  Andrew Rabinovich

IPC: G06N3/08 ,  G06T7/30 ,  G06T3/00 ,  G06T7/12 ,  G06T7/174 ,  G06F17/16 ,  G06K9/46 ,  G06T3/40

Abstract: A method for training a neural network includes receiving a plurality of images and, for each individual image of the plurality of images, generating a training triplet including a subset of the individual image, a subset of a transformed image, and a homography based on the subset of the individual image and the subset of the transformed image. The method also includes, for each individual image, generating, by the neural network, an estimated homography based on the subset of the individual image and the subset of the transformed image, comparing the estimated homography to the homography, and modifying the neural network based on the comparison.