公开(公告)号：US11657286B2

公开(公告)日：2023-05-23

申请号：US17183021

申请日：2021-02-23

Applicant: MAGIC LEAP, INC.

Inventor： Andrew Rabinovich ,  Vijay Badrinarayanan ,  Daniel DeTone ,  Srivignesh Rajendran ,  Douglas Bertram Lee ,  Tomasz Malisiewicz

IPC: G06K9/00 ,  G06V30/194 ,  G06N3/082 ,  G06V10/44 ,  G06F18/24 ,  G06F18/2413 ,  G06N3/045

CPC classification number: G06V30/194 ,  G06F18/24 ,  G06F18/24137 ,  G06N3/045 ,  G06N3/082 ,  G06V10/454

Abstract: The present disclosure provides an improved approach to implement structure learning of neural networks by exploiting correlations in the data/problem the networks aim to solve. A greedy approach is described that finds bottlenecks of information gain from the bottom convolutional layers all the way to the fully connected layers. Rather than simply making the architecture deeper, additional computation and capacitance is only added where it is required.

公开(公告)号：US11537894B2

公开(公告)日：2022-12-27

申请号：US17179226

申请日：2021-02-18

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Daniel DeTone ,  Tomasz Jan Malisiewicz

IPC: G06N3/08 ,  G06N3/04 ,  G06T7/00 ,  G06K9/62 ,  G06V10/44 ,  G06V10/46 ,  G06V20/10

Abstract: Systems, devices, and methods for training a neural network and performing image interest point detection and description using the neural network. The neural network may include an interest point detector subnetwork and a descriptor subnetwork. An optical device may include at least one camera for capturing a first image and a second image. A first set of interest points and a first descriptor may be calculated using the neural network based on the first image, and a second set of interest points and a second descriptor may be calculated using the neural network based on the second image. A homography between the first image and the second image may be determined based on the first and second sets of interest points and the first and second descriptors. The optical device may adjust virtual image light being projected onto an eyepiece based on the homography.

公开(公告)号：US11238606B2

公开(公告)日：2022-02-01

申请号：US16895878

申请日：2020-06-08

Applicant: Magic Leap, Inc.

Inventor： Daniel DeTone ,  Tomasz Jan Malisiewicz ,  Andrew Rabinovich

IPC: G06T7/593 ,  G06K9/00 ,  G06K9/62 ,  G06T7/33 ,  G06T7/35 ,  G06T7/579 ,  G06K9/46

Abstract: Augmented reality devices and methods for computing a homography based on two images. One method may include receiving a first image based on a first camera pose and a second image based on a second camera pose, generating a first point cloud based on the first image and a second point cloud based on the second image, providing the first point cloud and the second point cloud to a neural network, and generating, by the neural network, the homography based on the first point cloud and the second point cloud. The neural network may be trained by generating a plurality of points, determining a 3D trajectory, sampling the 3D trajectory to obtain camera poses viewing the points, projecting the points onto 2D planes, comparing a generated homography using the projected points to the ground-truth homography and modifying the neural network based on the comparison.

公开(公告)号：US20210365785A1

公开(公告)日：2021-11-25

申请号：US17341079

申请日：2021-06-07

Applicant: Magic Leap, Inc.

Inventor： Daniel DeTone ,  Tomasz Malisiewicz ,  Andrew Rabinovich

IPC: G06N3/08 ,  G06T7/30 ,  G06T3/00 ,  G06T7/12 ,  G06T7/174 ,  G06F17/16 ,  G06K9/46 ,  G06T3/40

Abstract: A method for training a neural network includes receiving a plurality of images and, for each individual image of the plurality of images, generating a training triplet including a subset of the individual image, a subset of a transformed image, and a homography based on the subset of the individual image and the subset of the transformed image. The method also includes, for each individual image, generating, by the neural network, an estimated homography based on the subset of the individual image and the subset of the transformed image, comparing the estimated homography to the homography, and modifying the neural network based on the comparison.

公开(公告)号：US10726570B2

公开(公告)日：2020-07-28

申请号：US16020541

申请日：2018-06-27

Applicant: Magic Leap, Inc.

Inventor： Daniel DeTone ,  Tomasz Jan Malisiewicz ,  Andrew Rabinovich

IPC: G06T7/593 ,  G06K9/62 ,  G06T7/33 ,  G06T7/35 ,  G06K9/00 ,  G06T7/579 ,  G06K9/46

Abstract: Augmented reality devices and methods for computing a homography based on two images. One method may include receiving a first image based on a first camera pose and a second image based on a second camera pose, generating a first point cloud based on the first image and a second point cloud based on the second image, providing the first point cloud and the second point cloud to a neural network, and generating, by the neural network, the homography based on the first point cloud and the second point cloud. The neural network may be trained by generating a plurality of points, determining a 3D trajectory, sampling the 3D trajectory to obtain camera poses viewing the points, projecting the points onto 2D planes, comparing a generated homography using the projected points to the ground-truth homography and modifying the neural network based on the comparison.

公开(公告)号：US10402649B2

公开(公告)日：2019-09-03

申请号：US15683664

申请日：2017-08-22

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Tomasz Jan Malisiewicz ,  Daniel DeTone

IPC: G06F1/16 ,  G06F3/01 ,  G06K9/00 ,  G06K9/62 ,  G06N3/00 ,  G06N3/04 ,  G06N3/08 ,  G06N5/00 ,  G06N7/00 ,  A63F13/00 ,  G02B27/01 ,  A63F13/211 ,  A63F13/212 ,  A63F13/213 ,  A63F13/428 ,  G06F3/0338 ,  G06F3/0346 ,  G06F3/0484

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

公开(公告)号：US10255529B2

公开(公告)日：2019-04-09

申请号：US15457990

申请日：2017-03-13

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Vijay Badrinarayanan ,  Daniel DeTone ,  Srivignesh Rajendran ,  Douglas Bertram Lee ,  Tomasz Malisiewicz

IPC: G06K9/00 ,  G06K9/66 ,  G06K9/62 ,  G06K9/46

Abstract: The present disclosure provides an improved approach to implement structure learning of neural networks by exploiting correlations in the data/problem the networks aim to solve. A greedy approach is described that finds bottlenecks of information gain from the bottom convolutional layers all the way to the fully connected layers. Rather than simply making the architecture deeper, additional computation and capacitance is only added where it is required.

公开(公告)号：US20180053056A1

公开(公告)日：2018-02-22

申请号：US15683664

申请日：2017-08-22

Applicant: Magic Leap, Inc.

Inventor： Andrew Rabinovich ,  Tomasz Jan Malisiewicz ,  Daniel DeTone

IPC: G06K9/00 ,  G06K9/62 ,  G06N3/08 ,  G06N3/04

CPC classification number: G06K9/00671 ,  A63F13/00 ,  A63F13/211 ,  A63F13/212 ,  A63F13/213 ,  A63F13/428 ,  G02B27/017 ,  G06F1/163 ,  G06F3/011 ,  G06F3/0338 ,  G06F3/0346 ,  G06F3/04842 ,  G06K9/00255 ,  G06K9/00288 ,  G06K9/4628 ,  G06K9/6256 ,  G06K9/627 ,  G06N3/006 ,  G06N3/04 ,  G06N3/0445 ,  G06N3/0454 ,  G06N3/08 ,  G06N5/003 ,  G06N7/005

Abstract: A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

公开(公告)号：US20170337470A1

公开(公告)日：2017-11-23

申请号：US15600545

申请日：2017-05-19

Applicant: Magic Leap, Inc.

Inventor： Daniel DeTone ,  Tomasz Malisiewicz ,  Andrew Rabinovich

IPC: G06N3/08 ,  G06K9/46 ,  G06F17/16 ,  G06T7/12 ,  G06T3/40 ,  G06T7/174

CPC classification number: G06N3/08 ,  G06F17/16 ,  G06K9/46 ,  G06T3/0068 ,  G06T3/403 ,  G06T7/12 ,  G06T7/174 ,  G06T7/30 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/20164

Abstract: A method for generating inputs for a neural network based on an image includes receiving the image, identifying a position within the image, and identifying a subset of the image at the position. The subset of the image is defined by a first set of corners. The method also includes perturbing at least one of the first set of corners to form a second set of corners. The second set of corners defines a modified subset of the image. The method further includes determining a homography based on a comparison between the subset of the image and the modified subset of the image, generating a transformed image by applying the homography to the image, and identifying a subset of the transformed image at the position.