公开(公告)号：US11062209B2

公开(公告)日：2021-07-13

申请号：US16588505

申请日：2019-09-30

Applicant: Magic Leap, Inc.

Inventor： Daniel DeTone ,  Tomasz Malisiewicz ,  Andrew Rabinovich

IPC: G06K9/00 ,  G06N3/08 ,  G06T7/30 ,  G06T3/00 ,  G06T7/12 ,  G06T7/174 ,  G06F17/16 ,  G06K9/46 ,  G06T3/40

Abstract: A method for training a neural network includes receiving a plurality of images and, for each individual image of the plurality of images, generating a training triplet including a subset of the individual image, a subset of a transformed image, and a homography based on the subset of the individual image and the subset of the transformed image. The method also includes, for each individual image, generating, by the neural network, an estimated homography based on the subset of the individual image and the subset of the transformed image, comparing the estimated homography to the homography, and modifying the neural network based on the comparison.

公开(公告)号：US20170262737A1

公开(公告)日：2017-09-14

申请号：US15457990

申请日：2017-03-13

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Vijay Badrinarayanan ,  Daniel DeTone ,  Srivignesh Rajendran ,  Douglas Bertram Lee ,  Tomasz Malisiewicz

IPC: G06K9/66 ,  G06K9/62

CPC classification number: G06K9/66 ,  G06K9/4628 ,  G06K9/6267 ,  G06K9/6272

Abstract: The present disclosure provides an improved approach to implement structure learning of neural networks by exploiting correlations in the data/problem the networks aim to solve. A greedy approach is described that finds bottlenecks of information gain from the bottom convolutional layers all the way to the fully connected layers. Rather than simply making the architecture deeper, additional computation and capacitance is only added where it is required.

公开(公告)号：US11593654B2

公开(公告)日：2023-02-28

申请号：US17341079

申请日：2021-06-07

Applicant: Magic Leap, Inc.

Inventor： Daniel DeTone ,  Tomasz Malisiewicz ,  Andrew Rabinovich

IPC: G06K9/00 ,  G06N3/08 ,  G06T7/30 ,  G06T3/00 ,  G06V10/40 ,  G06V10/82 ,  G06T7/12 ,  G06T7/174 ,  G06F17/16 ,  G06T3/40

Abstract: A method for training a neural network includes receiving a plurality of images and, for each individual image of the plurality of images, generating a training triplet including a subset of the individual image, a subset of a transformed image, and a homography based on the subset of the individual image and the subset of the transformed image. The method also includes, for each individual image, generating, by the neural network, an estimated homography based on the subset of the individual image and the subset of the transformed image, comparing the estimated homography to the homography, and modifying the neural network based on the comparison.

公开(公告)号：US20210350566A1

公开(公告)日：2021-11-11

申请号：US17288877

申请日：2019-11-14

Applicant: Magic Leap, Inc.

Inventor： Danying Hu ,  Daniel DeTone ,  Tomasz Jan Malisiewicz

IPC: G06T7/70 ,  G06T7/80 ,  G06T3/40 ,  G06T9/00 ,  G06T19/00 ,  G06N3/08 ,  G06N3/04 ,  G02B27/01

Abstract: A deep neural network provides real-time pose estimation by combining two custom deep neural networks, a location classifier and an ID classifier, with a pose estimation algorithm to achieve a 6D0F location of a fiducial marker. The locations may be further refined into subpixel coordinates using another deep neural network. The networks may be trained using a combination of auto-labeled videos of the target marker, synthetic subpixel corner data, and/or extreme data augmentation. The deep neural network provides improved pose estimations particularly in challenging low-light, high-motion, and/or high-blur scenarios.

公开(公告)号：US20210241114A1

公开(公告)日：2021-08-05

申请号：US17179226

申请日：2021-02-18

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Daniel DeTone ,  Tomasz Jan Malisiewicz

IPC: G06N3/08 ,  G06N3/04 ,  G06T7/00 ,  G06K9/46 ,  G06K9/62 ,  G06K9/00

Abstract: Systems, devices, and methods for training a neural network and performing image interest point detection and description using the neural network. The neural network may include an interest point detector subnetwork and a descriptor subnetwork. An optical device may include at least one camera for capturing a first image and a second image. A first set of interest points and a first descriptor may be calculated using the neural network based on the first image, and a second set of interest points and a second descriptor may be calculated using the neural network based on the second image. A homography between the first image and the second image may be determined based on the first and second sets of interest points and the first and second descriptors. The optical device may adjust virtual image light being projected onto an eyepiece based on the homography.

公开(公告)号：US20210182636A1

公开(公告)日：2021-06-17

申请号：US17183021

申请日：2021-02-23

Applicant: MAGIC LEAP, INC.

Inventor： Andrew Rabinovich ,  Vijay Badrinarayanan ,  Daniel DeTone ,  Srivignesh Rajendran ,  Douglas Bertram Lee ,  Tomasz Malisiewicz

IPC: G06K9/66 ,  G06K9/62 ,  G06K9/46 ,  G06N3/04 ,  G06N3/08

Abstract: The present disclosure provides an improved approach to implement structure learning of neural networks by exploiting correlations in the data/problem the networks aim to solve. A greedy approach is described that finds bottlenecks of information gain from the bottom convolutional layers all the way to the fully connected layers. Rather than simply making the architecture deeper, additional computation and capacitance is only added where it is required.

公开(公告)号：US20190005670A1

公开(公告)日：2019-01-03

申请号：US16020541

申请日：2018-06-27

Applicant: Magic Leap, Inc.

Inventor： Daniel DeTone ,  Tomasz Jan Malisiewicz ,  Andrew Rabinovich

IPC: G06T7/593 ,  G06K9/62

Abstract: Augmented reality devices and methods for computing a homography based on two images. One method may include receiving a first image based on a first camera pose and a second image based on a second camera pose, generating a first point cloud based on the first image and a second point cloud based on the second image, providing the first point cloud and the second point cloud to a neural network, and generating, by the neural network, the homography based on the first point cloud and the second point cloud. The neural network may be trained by generating a plurality of points, determining a 3D trajectory, sampling the 3D trajectory to obtain camera poses viewing the points, projecting the points onto 2D planes, comparing a generated homography using the projected points to the ground-truth homography and modifying the neural network based on the comparison.

公开(公告)号：US12033081B2

公开(公告)日：2024-07-09

申请号：US17098043

申请日：2020-11-13

Applicant: Magic Leap, Inc.

Inventor： Paul-Edouard Sarlin ,  Daniel DeTone ,  Tomasz Jan Malisiewicz ,  Andrew Rabinovich

IPC: G06N3/084 ,  G06F18/22 ,  G06N3/04 ,  G06N3/08 ,  G06V10/75 ,  G06V10/82 ,  G06V20/00 ,  G06V30/196

CPC classification number: G06N3/084 ,  G06F18/22 ,  G06N3/04 ,  G06N3/08 ,  G06V10/757 ,  G06V10/82 ,  G06V20/36 ,  G06V30/1988

Abstract: The description relates the feature matching. Our approach establishes pointwise correspondences between challenging image pairs. It takes off-the-shelf local features as input and uses an attentional graph neural network to solve an assignment optimization problem. The deep middle-end matcher acts as a middle-end and handles partial point visibility and occlusion elegantly, producing a partial assignment matrix.

公开(公告)号：US11797078B2

公开(公告)日：2023-10-24

申请号：US17472151

申请日：2021-09-10

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Tomasz Jan Malisiewicz ,  Daniel DeTone

IPC: G09G5/00 ,  G06F3/01 ,  A63F13/211 ,  A63F13/212 ,  A63F13/213 ,  A63F13/00 ,  G06F1/16 ,  G06F3/0338 ,  G06F3/0346 ,  G06F3/04842 ,  G06N3/006 ,  G06V20/20 ,  G06V10/44 ,  G06V40/16 ,  G06F18/2413 ,  G06N3/044 ,  G06N3/045 ,  G06V10/764 ,  G06V10/82 ,  G06N3/04 ,  G06N3/08 ,  A63F13/428 ,  G02B27/01 ,  G06F18/214 ,  G06N5/01 ,  G06N7/01

CPC classification number: G06F3/011 ,  A63F13/00 ,  A63F13/211 ,  A63F13/212 ,  A63F13/213 ,  G06F1/163 ,  G06F3/0338 ,  G06F3/0346 ,  G06F3/04842 ,  G06F18/2413 ,  G06N3/006 ,  G06N3/04 ,  G06N3/044 ,  G06N3/045 ,  G06N3/08 ,  G06V10/454 ,  G06V10/764 ,  G06V10/82 ,  G06V20/20 ,  G06V40/166 ,  G06V40/172 ,  A63F13/428 ,  G02B27/017 ,  G06F18/214 ,  G06N5/01 ,  G06N7/01

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

公开(公告)号：US20200302628A1

公开(公告)日：2020-09-24

申请号：US16895878

申请日：2020-06-08

Applicant: Magic Leap, Inc.

Inventor： Daniel DeTone ,  Tomasz Jan Malisiewicz ,  Andrew Rabinovich

IPC: G06T7/593 ,  G06K9/62 ,  G06T7/33 ,  G06T7/35 ,  G06K9/00 ,  G06T7/579 ,  G06K9/46

Abstract: Augmented reality devices and methods for computing a homography based on two images. One method may include receiving a first image based on a first camera pose and a second image based on a second camera pose, generating a first point cloud based on the first image and a second point cloud based on the second image, providing the first point cloud and the second point cloud to a neural network, and generating, by the neural network, the homography based on the first point cloud and the second point cloud. The neural network may be trained by generating a plurality of points, determining a 3D trajectory, sampling the 3D trajectory to obtain camera poses viewing the points, projecting the points onto 2D planes, comparing a generated homography using the projected points to the ground-truth homography and modifying the neural network based on the comparison.