公开(公告)号：US12020167B2

公开(公告)日：2024-06-25

申请号：US17051982

申请日：2019-05-15

Applicant: Magic Leap, Inc.

Inventor： Ayan Tuhinendu Sinha ,  Andrew Rabinovich ,  Zhao Chen ,  Vijay Badrinarayanan

IPC: G06N3/08 ,  G06N3/045 ,  G06N3/084 ,  G06N3/088

CPC classification number: G06N3/088 ,  G06N3/045 ,  G06N3/084

Abstract: Systems and methods for gradient adversarial training of a neural network are disclosed. In one aspect of gradient adversarial training, an auxiliary neural network can be trained to classify a gradient tensor that is evaluated during backpropagation in a main neural network that provides a desired task output. The main neural network can serve as an adversary to the auxiliary network in addition to a standard task-based training procedure. The auxiliary neural network can pass an adversarial gradient signal back to the main neural network, which can use this signal to regularize the weight tensors in the main neural network. Gradient adversarial training of the neural network can provide improved gradient tensors in the main network. Gradient adversarial techniques can be used to train multitask networks, knowledge distillation networks, and adversarial defense networks.

公开(公告)号：US20230394315A1

公开(公告)日：2023-12-07

申请号：US18454680

申请日：2023-08-23

Applicant: Magic Leap, Inc.

Inventor： Chen-Yu Lee ,  Vijay Badrinarayanan ,  Tomasz Jan Malisiewicz ,  Andrew Rabinovich

IPC: G06N3/084 ,  G06N3/04 ,  G06V20/00 ,  G06V10/44 ,  G06V10/94 ,  G06V20/20 ,  G06F18/214 ,  G06F18/2413 ,  G06N3/044 ,  G06N3/045 ,  G06V10/764 ,  G06V10/774 ,  G06V10/82

CPC classification number: G06N3/084 ,  G06N3/04 ,  G06V20/36 ,  G06V10/44 ,  G06V10/95 ,  G06V10/454 ,  G06V20/20 ,  G06F18/214 ,  G06F18/2413 ,  G06N3/044 ,  G06N3/045 ,  G06V10/764 ,  G06V10/774 ,  G06V10/82 ,  G06N3/082

Abstract: Systems and methods for estimating a layout of a room are disclosed. The room layout can comprise the location of a floor, one or more walls, and a ceiling. In one aspect, a neural network can analyze an image of a portion of a room to determine the room layout. The neural network can comprise a convolutional neural network having an encoder sub-network, a decoder sub-network, and a side sub-network. The neural network can determine a three-dimensional room layout using two-dimensional ordered keypoints associated with a room type. The room layout can be used in applications such as augmented or mixed reality, robotics, autonomous indoor navigation, etc.

公开(公告)号：US11797860B2

公开(公告)日：2023-10-24

申请号：US17717696

申请日：2022-04-11

Applicant: Magic Leap, Inc.

Inventor： Tomasz Jan Malisiewicz ,  Andrew Rabinovich ,  Vijay Badrinarayanan ,  Debidatta Dwibedi

IPC: G06T7/70 ,  G06N3/084 ,  G06V10/44 ,  G06V20/64 ,  G06F18/2413 ,  G06N3/044 ,  G06N3/045 ,  G06V30/19 ,  G06V10/82 ,  G06T7/11 ,  G06N3/08

CPC classification number: G06N3/084 ,  G06F18/24133 ,  G06N3/044 ,  G06N3/045 ,  G06N3/08 ,  G06T7/11 ,  G06T7/70 ,  G06V10/454 ,  G06V10/82 ,  G06V20/64 ,  G06V30/19173 ,  G06T2210/12

Abstract: Systems and methods for cuboid detection and keypoint localization in images are disclosed. In one aspect, a deep cuboid detector can be used for simultaneous cuboid detection and keypoint localization in monocular images. The deep cuboid detector can include a plurality of convolutional layers and non-convolutional layers of a trained convolution neural network for determining a convolutional feature map from an input image. A region proposal network of the deep cuboid detector can determine a bounding box surrounding a cuboid in the image using the convolutional feature map. The pooling layer and regressor layers of the deep cuboid detector can implement iterative feature pooling for determining a refined bounding box and a parameterized representation of the cuboid.

公开(公告)号：US11682127B2

公开(公告)日：2023-06-20

申请号：US17018940

申请日：2020-09-11

Applicant: Magic Leap, Inc.

Inventor： Vijay Badrinarayanan ,  Zhao Chen ,  Andrew Rabinovich

IPC: G06T7/50 ,  G06N3/08

CPC classification number: G06T7/50 ,  G06N3/08 ,  G06T2207/10024 ,  G06T2207/20081 ,  G06T2207/20084

Abstract: Systems and methods are disclosed for training and using neural networks for computing depth maps. One method for training the neural network includes providing an image input to the neural network. The image input may include a camera image of a training scene. The method may also include providing a depth input to the neural network. The depth input may be based on a high-density depth map of the training scene and a sampling mask. The method may further include generating, using the neural network, a computed depth map of the training scene based on the image input and the depth input. The method may further include modifying the neural network based on an error between the computed depth map and the high-density depth map.

公开(公告)号：US11657286B2

公开(公告)日：2023-05-23

申请号：US17183021

申请日：2021-02-23

Applicant: MAGIC LEAP, INC.

Inventor： Andrew Rabinovich ,  Vijay Badrinarayanan ,  Daniel DeTone ,  Srivignesh Rajendran ,  Douglas Bertram Lee ,  Tomasz Malisiewicz

IPC: G06K9/00 ,  G06V30/194 ,  G06N3/082 ,  G06V10/44 ,  G06F18/24 ,  G06F18/2413 ,  G06N3/045

CPC classification number: G06V30/194 ,  G06F18/24 ,  G06F18/24137 ,  G06N3/045 ,  G06N3/082 ,  G06V10/454

Abstract: The present disclosure provides an improved approach to implement structure learning of neural networks by exploiting correlations in the data/problem the networks aim to solve. A greedy approach is described that finds bottlenecks of information gain from the bottom convolutional layers all the way to the fully connected layers. Rather than simply making the architecture deeper, additional computation and capacitance is only added where it is required.

公开(公告)号：US20220244781A1

公开(公告)日：2022-08-04

申请号：US17674724

申请日：2022-02-17

Applicant: Magic Leap, Inc.

Inventor： Zhengyang Wu ,  Srivignesh Rajendran ,  Tarrence van As ,  Joelle Zimmermann ,  Vijay Badrinarayanan ,  Andrew Rabinovich

IPC: G06F3/01 ,  G06V10/26 ,  G06V40/18 ,  G06V10/82 ,  G06V10/774 ,  G02B27/00

Abstract: Techniques related to the computation of gaze vectors of users of wearable devices are disclosed. A neural network may be trained through first and second training steps. The neural network may include a set of feature encoding layers and a plurality of sets of task-specific layers that each operate on an output of the set of feature encoding layers. During the first training step, a first image of a first eye may be provided to the neural network, eye segmentation data may be generated using the neural network, and the set of feature encoding layers may be trained. During the second training step, a second image of a second eye may be provided to the neural network, network output data may be generated using the neural network, and the plurality of sets of task-specific layers may be trained.

公开(公告)号：US20210327085A1

公开(公告)日：2021-10-21

申请号：US17221250

申请日：2021-04-02

Applicant: Magic Leap, Inc.

Inventor： Adrian Kaehler ,  Douglas Bertram Lee ,  Vijay Badrinarayanan

IPC: G06T7/70 ,  G06T7/20 ,  G06F3/01 ,  G06F3/0481 ,  G06N3/02 ,  G02B27/01 ,  G06F3/0346 ,  G06T7/246 ,  G06F1/16 ,  G02B27/00

Abstract: Disclosed herein is a wearable display system for capturing retraining eye images of an eye of a user for retraining a neural network for eye tracking. The system captures retraining eye images using an image capture device when user interface (UI) events occur with respect to UI devices displayed at display locations of a display. The system can generate a retraining set comprising the retraining eye images and eye poses of the eye of the user in the retraining eye images (e.g., related to the display locations of the UI devices) and obtain a retrained neural network that is retrained using the retraining set.

公开(公告)号：US20210133506A1

公开(公告)日：2021-05-06

申请号：US17148249

申请日：2021-01-13

Applicant: Magic Leap, Inc.

Inventor： Adrian Kaehler ,  Gary R. Bradski ,  Vijay Badrinarayanan

IPC: G06K9/62 ,  G06N3/04 ,  G06N3/10 ,  G06K9/00 ,  G06F3/01 ,  G06T7/70 ,  G06F3/0346 ,  G06F3/0481 ,  G06N3/08 ,  G06T19/00

Abstract: Disclosed herein are examples of a wearable display system capable of determining a user interface (UI) event with respect to a virtual UI device (e.g., a button) and a pointer (e.g., a finger or a stylus) using a neural network. The wearable display system can render a representation of the UI device onto an image of the pointer captured when the virtual UI device is shown to the user and the user uses the pointer to interact with the virtual UI device. The representation of the UI device can include concentric shapes (or shapes with similar or the same centers of gravity) of high contrast. The neural network can be trained using training images with representations of virtual UI devices and pointers.

公开(公告)号：US10922583B2

公开(公告)日：2021-02-16

申请号：US15994599

申请日：2018-05-31

Applicant: Magic Leap, Inc.

Inventor： Adrian Kaehler ,  Gary R. Bradski ,  Vijay Badrinarayanan

IPC: G06K9/62 ,  G06N3/04 ,  G06N3/10 ,  G06K9/00 ,  G06F3/01 ,  G06T7/70 ,  G06F3/0346 ,  G06F3/0481 ,  G06N3/08 ,  G06T19/00 ,  G06K9/46 ,  G06N7/00 ,  G06N5/02 ,  G06N5/00

Abstract: An example wearable display system is capable of determining a user interface (UI) event with respect to a virtual UI device (e.g., a button) and a pointer (e.g., a finger or a stylus) using a neural network. The wearable display system can render a representation of the UI device onto an image of the pointer captured when the virtual UI device is shown to the user and the user uses the pointer to interact with the virtual UI device. The representation of the UI device can include concentric shapes (or shapes with similar or the same centers of gravity) of high contrast. The neural network can be trained using training images with representations of virtual UI devices and pointers.

公开(公告)号：US10719951B2

公开(公告)日：2020-07-21

申请号：US16134600

申请日：2018-09-18

Applicant: Magic Leap, Inc.

Inventor： Adrian Kaehler ,  Douglas Bertram Lee ,  Vijay Badrinarayanan

IPC: G06K9/46 ,  G06T7/70 ,  G06T7/20 ,  G06F3/01 ,  G06F3/0481 ,  G06N3/02 ,  G02B27/01 ,  G06F3/0346 ,  G06T7/246 ,  G06F1/16 ,  G02B27/00 ,  G06T19/00

Abstract: Disclosed herein is a wearable display system for capturing retraining eye images of an eye of a user for retraining a neural network for eye tracking. The system captures retraining eye images using an image capture device when user interface (UI) events occur with respect to UI devices displayed at display locations of a display. The system can generate a retraining set comprising the retraining eye images and eye poses of the eye of the user in the retraining eye images (e.g., related to the display locations of the UI devices) and obtain a retrained neural network that is retrained using the retraining set.