公开(公告)号：US20240330691A1

公开(公告)日：2024-10-03

申请号：US18661377

申请日：2024-05-10

Applicant: Magic Leap, Inc.

Inventor： Ayan Tuhinendu Sinha ,  Andrew Rabinovich ,  Zhao Chen ,  Vijay Badrinarayanan

IPC: G06N3/088 ,  G06N3/045 ,  G06N3/084

CPC classification number: G06N3/088 ,  G06N3/045 ,  G06N3/084

Abstract: Systems and methods for gradient adversarial training of a neural network are disclosed. In one aspect of gradient adversarial training, an auxiliary neural network can be trained to classify a gradient tensor that is evaluated during backpropagation in a main neural network that provides a desired task output. The main neural network can serve as an adversary to the auxiliary network in addition to a standard task-based training procedure. The auxiliary neural network can pass an adversarial gradient signal back to the main neural network, which can use this signal to regularize the weight tensors in the main neural network. Gradient adversarial training of the neural network can provide improved gradient tensors in the main network. Gradient adversarial techniques can be used to train multitask networks, knowledge distillation networks, and adversarial defense networks.

公开(公告)号：US11537895B2

公开(公告)日：2022-12-27

申请号：US16169840

申请日：2018-10-24

Applicant: Magic Leap, Inc.

Inventor： Zhao Chen ,  Vijay Badrinarayanan ,  Andrew Rabinovich

IPC: G06N3/08 ,  G06N3/04 ,  G06N20/00

Abstract: Systems and methods for training a multitask network is disclosed. In one aspect, training the multitask network includes determining a gradient norm of a single-task loss adjusted by a task weight for each task, with respect to network weights of the multitask network, and a relative training rate for the task based on the single-task loss for the task. Subsequently, a gradient loss function, comprising a difference between (1) the determined gradient norm for each task and (2) a corresponding target gradient norm, can be determined. An updated task weight for the task can be determined and used in the next iteration of training the multitask network, using a gradient of the gradient loss function with respect to the task weight for the task.

公开(公告)号：US20210192357A1

公开(公告)日：2021-06-24

申请号：US17051982

申请日：2019-05-15

Applicant: Magic Leap, Inc.

Inventor： Ayan Tuhinendu Sinha ,  Andrew Rabinovich ,  Zhao Chen ,  Vijay Badrinarayanan

IPC: G06N3/08 ,  G06N3/04

Abstract: Systems and methods for gradient adversarial training of a neural network are disclosed. In one aspect of gradient adversarial training, an auxiliary neural network can be trained to classify a gradient tensor that is evaluated during backpropagation in a main neural network that provides a desired task output. The main neural network can serve as an adversary to the auxiliary network in addition to a standard task-based training procedure. The auxiliary neural network can pass an adversarial gradient signal back to the main neural network, which can use this signal to regularize the weight tensors in the main neural network. Gradient adversarial training of the neural network can provide improved gradient tensors in the main network. Gradient adversarial techniques can be used to train multitask networks, knowledge distillation networks, and adversarial defense networks.

公开(公告)号：US20190130275A1

公开(公告)日：2019-05-02

申请号：US16169840

申请日：2018-10-24

Applicant: Magic Leap, Inc.

Inventor： Zhao Chen ,  Vijay Badrinarayanan ,  Andrew Rabinovich

IPC: G06N3/08 ,  G06N3/04 ,  G06F15/18

Abstract: Systems and methods for training a multitask network is disclosed. In one aspect, training the multitask network includes determining a gradient norm of a single-task loss adjusted by a task weight for each task, with respect to network weights of the multitask network, and a relative training rate for the task based on the single-task loss for the task. Subsequently, a gradient loss function, comprising a difference between (1) the determined gradient norm for each task and (2) a corresponding target gradient norm, can be determined. An updated task weight for the task can be determined and used in the next iteration of training the multitask network, using a gradient of the gradient loss function with respect to the task weight for the task.

公开(公告)号：US11128854B2

公开(公告)日：2021-09-21

申请号：US16352522

申请日：2019-03-13

Applicant: Magic Leap, Inc.

Inventor： Vijay Badrinarayanan ,  Zhao Chen ,  Andrew Rabinovich ,  Elad Joseph

IPC: G06K9/00 ,  H04N13/271 ,  H04N13/128 ,  G06T7/593 ,  H04N13/00 ,  G06T19/00

Abstract: Systems and methods are disclosed for computing depth maps. One method includes capturing, using a camera, a camera image of a runtime scene. The method may also include analyzing the camera image of the runtime scene to determine a plurality of target sampling points at which to capture depth of the runtime scene. The method may further include adjusting a setting associated with a low-density depth sensor based on the plurality of target sampling points. The method may further include capturing, using the low-density depth sensor, a low-density depth map of the runtime scene at the plurality of target sampling points. The method may further include generating a computed depth map of the runtime scene based on the camera image of the runtime scene and the low-density depth map of the runtime scene.

公开(公告)号：US12020167B2

公开(公告)日：2024-06-25

申请号：US17051982

申请日：2019-05-15

Applicant: Magic Leap, Inc.

Inventor： Ayan Tuhinendu Sinha ,  Andrew Rabinovich ,  Zhao Chen ,  Vijay Badrinarayanan

IPC: G06N3/08 ,  G06N3/045 ,  G06N3/084 ,  G06N3/088

CPC classification number: G06N3/088 ,  G06N3/045 ,  G06N3/084

Abstract: Systems and methods for gradient adversarial training of a neural network are disclosed. In one aspect of gradient adversarial training, an auxiliary neural network can be trained to classify a gradient tensor that is evaluated during backpropagation in a main neural network that provides a desired task output. The main neural network can serve as an adversary to the auxiliary network in addition to a standard task-based training procedure. The auxiliary neural network can pass an adversarial gradient signal back to the main neural network, which can use this signal to regularize the weight tensors in the main neural network. Gradient adversarial training of the neural network can provide improved gradient tensors in the main network. Gradient adversarial techniques can be used to train multitask networks, knowledge distillation networks, and adversarial defense networks.

公开(公告)号：US11682127B2

公开(公告)日：2023-06-20

申请号：US17018940

申请日：2020-09-11

Applicant: Magic Leap, Inc.

Inventor： Vijay Badrinarayanan ,  Zhao Chen ,  Andrew Rabinovich

IPC: G06T7/50 ,  G06N3/08

CPC classification number: G06T7/50 ,  G06N3/08 ,  G06T2207/10024 ,  G06T2207/20081 ,  G06T2207/20084

Abstract: Systems and methods are disclosed for training and using neural networks for computing depth maps. One method for training the neural network includes providing an image input to the neural network. The image input may include a camera image of a training scene. The method may also include providing a depth input to the neural network. The depth input may be based on a high-density depth map of the training scene and a sampling mask. The method may further include generating, using the neural network, a computed depth map of the training scene based on the image input and the depth input. The method may further include modifying the neural network based on an error between the computed depth map and the high-density depth map.

公开(公告)号：US11600049B2

公开(公告)日：2023-03-07

申请号：US16856980

申请日：2020-04-23

Applicant: Magic Leap, Inc.

Inventor： Zhao Chen ,  Ameya Pramod Phalak ,  Vijay Badrinarayanan

IPC: G06T19/00 ,  G06T7/50 ,  G06T7/13

Abstract: Techniques for estimating a perimeter of a room environment at least partially enclosed by a set of adjoining walls using posed images are disclosed. A set of images and a set of poses are obtained. A depth map is generated based on the set of images and the set of poses. A set of wall segmentation maps are generated based on the set of images, each of the set of wall segmentation maps indicating a target region of a corresponding image that contains the set of adjoining walls. A point cloud is generated based on the depth map and the set of wall segmentation maps, the point cloud including a plurality of points that are sampled along portions of the depth map that align with the target region. The perimeter of the environment along the set of adjoining walls is estimated based on the point cloud.

公开(公告)号：US20200342674A1

公开(公告)日：2020-10-29

申请号：US16856980

申请日：2020-04-23

Applicant: Magic Leap, Inc.

Inventor： Zhao Chen ,  Ameya Pramod Phalak ,  Vijay Badrinarayanan

IPC: G06T19/00 ,  G06T7/50 ,  G06T7/13

Abstract: Techniques for estimating a perimeter of a room environment at least partially enclosed by a set of adjoining walls using posed images are disclosed. A set of images and a set of poses are obtained. A depth map is generated based on the set of images and the set of poses. A set of wall segmentation maps are generated based on the set of images, each of the set of wall segmentation maps indicating a target region of a corresponding image that contains the set of adjoining walls. A point cloud is generated based on the depth map and the set of wall segmentation maps, the point cloud including a plurality of points that are sampled along portions of the depth map that align with the target region. The perimeter of the environment along the set of adjoining walls is estimated based on the point cloud.