公开(公告)号：US11704814B2

公开(公告)日：2023-07-18

申请号：US17319891

申请日：2021-05-13

Applicant: NVIDIA Corporation

Inventor： Nuri Murat Arar ,  Niranjan Avadhanam ,  Hairong Jiang ,  Nishant Puri ,  Rajath Shetty ,  Shagan Sah

IPC: G06T7/20 ,  G06N3/08 ,  G06K9/00 ,  G06V20/59 ,  G06V40/18 ,  G06F18/211 ,  G06F18/21

CPC classification number: G06T7/20 ,  G06F18/211 ,  G06F18/2178 ,  G06N3/08 ,  G06V20/59 ,  G06V40/18

Abstract: In various examples, an adaptive eye tracking machine learning model engine (“adaptive-model engine”) for an eye tracking system is described. The adaptive-model engine may include an eye tracking or gaze tracking development pipeline (“adaptive-model training pipeline”) that supports collecting data, training, optimizing, and deploying an adaptive eye tracking model that is a customized eye tracking model based on a set of features of an identified deployment environment. The adaptive-model engine supports ensembling the adaptive eye tracking model that may be trained on gaze vector estimation in surround environments and ensemble based on a plurality of eye tracking variant models and a plurality of facial landmark neural network metrics.

公开(公告)号：US20230206488A1

公开(公告)日：2023-06-29

申请号：US18182086

申请日：2023-03-10

Applicant: NVIDIA Corporation

Inventor： Nishant Puri

IPC: G06T7/73 ,  G06T7/20 ,  G06T7/00 ,  G06T7/70 ,  G06N20/00 ,  G06N3/08 ,  B60W30/08 ,  G06V20/56 ,  G06V40/16 ,  G06V40/18 ,  G06V10/764 ,  G06V10/82

CPC classification number: G06T7/73 ,  B60W30/08 ,  G06N3/08 ,  G06N20/00 ,  G06T7/0012 ,  G06T7/20 ,  G06T7/70 ,  G06V10/82 ,  G06V10/764 ,  G06V20/56 ,  G06V40/171 ,  G06V40/193 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/30041

Abstract: Machine learning systems and methods that determine gaze direction by using face orientation information, such as facial landmarks, to modify eye direction information determined from images of the subject’s eyes. System inputs include eye crops of the eyes of the subject, as well as face orientation information such as facial landmarks of the subject’s face in the input image. Facial orientation information, or facial landmark information, is used to determine a coarse prediction of gaze direction as well as to learn a context vector of features describing subject face pose. The context vector is then used to adaptively re-weight the eye direction features determined from the eye crops. The re-weighted features are then combined with the coarse gaze prediction to determine gaze direction.

公开(公告)号：US11636609B2

公开(公告)日：2023-04-25

申请号：US17010205

申请日：2020-09-02

Applicant: NVIDIA Corporation

Inventor： Nishant Puri

IPC: G06K9/46 ,  G06T7/20 ,  G06T7/00 ,  G06T7/70 ,  G06N20/00 ,  G06N3/08 ,  B60W30/08 ,  G06V20/56 ,  G06V40/16 ,  G06V40/18

Abstract: Machine learning systems and methods that determine gaze direction by using face orientation information, such as facial landmarks, to modify eye direction information determined from images of the subject's eyes. System inputs include eye crops of the eyes of the subject, as well as face orientation information such as facial landmarks of the subject's face in the input image. Facial orientation information, or facial landmark information, is used to determine a coarse prediction of gaze direction as well as to learn a context vector of features describing subject face pose. The context vector is then used to adaptively re-weight the eye direction features determined from the eye crops. The re-weighted features are then combined with the coarse gaze prediction to determine gaze direction.

公开(公告)号：US20230064049A1

公开(公告)日：2023-03-02

申请号：US17462833

申请日：2021-08-31

Applicant: Nvidia Corporation

Inventor： Sakthivel Sivaraman ,  Nishant Puri ,  Yuzhuo Ren ,  Atousa Torabi ,  Shubhadeep Das ,  Niranjan Avadhanam ,  Sumit Kumar Bhattacharya ,  Jason Roche

IPC: G06K9/00 ,  G06T7/73 ,  G06F3/01 ,  G06F16/632 ,  G06T15/06

Abstract: Interactions with virtual systems may be difficult when users inadvertently fail to provide sufficient information to proceed with their requests. Certain types of inputs, such as auditory inputs, may lack sufficient information to properly provide a response to the user. Additional information, such as image data, may enable user gestures or poses to supplement the auditory inputs to enable response generation without requesting additional information from users.

公开(公告)号：US20220366568A1

公开(公告)日：2022-11-17

申请号：US17319891

申请日：2021-05-13

Applicant: NVIDIA Corporation

Inventor： Nuri Murat Arar ,  Niranjan Avadhanam ,  Hairong Jiang ,  Nishant Puri ,  Rajath Shetty ,  Shagan Sah

IPC: G06T7/20 ,  G06K9/00 ,  G06K9/62 ,  G06N3/08

Abstract: In various examples, an adaptive eye tracking machine learning model engine (“adaptive-model engine”) for an eye tracking system is described. The adaptive-model engine may include an eye tracking or gaze tracking development pipeline (“adaptive-model training pipeline”) that supports collecting data, training, optimizing, and deploying an adaptive eye tracking model that is a customized eye tracking model based on a set of features of an identified deployment environment. The adaptive-model engine supports ensembling the adaptive eye tracking model that may be trained on gaze vector estimation in surround environments and ensemble based on a plurality of eye tracking variant models and a plurality of facial landmark neural network metrics.

公开(公告)号：US20220283638A1

公开(公告)日：2022-09-08

申请号：US17751548

申请日：2022-05-23

Applicant: NVIDIA Corporation

Inventor： Hairong Jiang ,  Nishant Puri ,  Niranjan Avadhanam ,  Nuri Murat Arar

IPC: G06F3/01 ,  G06V40/19

Abstract: Machine learning systems and methods that learn glare, and thus determine gaze direction in a manner more resilient to the effects of glare on input images. The machine learning systems have an isolated representation of glare, e.g., information on the locations of glare points in an image, as an explicit input, in addition to the image itself. In this manner, the machine learning systems explicitly consider glare while making a determination of gaze direction, thus producing more accurate results for images containing glare.

公开(公告)号：US20210182625A1

公开(公告)日：2021-06-17

申请号：US17004252

申请日：2020-08-27

Applicant: NVIDIA Corporation

Inventor： Nuri Murat Arar ,  Niranjan Avadhanam ,  Nishant Puri ,  Shagan Sah ,  Rajath Shetty ,  Sujay Yadawadkar ,  Pavlo Molchanov

IPC: G06K9/62 ,  G06K9/00

Abstract: Systems and methods for more accurate and robust determination of subject characteristics from an image of the subject. One or more machine learning models receive as input an image of a subject, and output both facial landmarks and associated confidence values. Confidence values represent the degrees to which portions of the subject's face corresponding to those landmarks are occluded, i.e., the amount of uncertainty in the position of each landmark location. These landmark points and their associated confidence values, and/or associated information, may then be input to another set of one or more machine learning models which may output any facial analysis quantity or quantities, such as the subject's gaze direction, head pose, drowsiness state, cognitive load, or distraction state.

公开(公告)号：US20210181837A1

公开(公告)日：2021-06-17

申请号：US16902737

申请日：2020-06-16

Applicant: NVIDIA Corporation

Inventor： Hairong Jiang ,  Nishant Puri ,  Niranjan Avadhanam ,  Nuri Murat Arar

IPC: G06F3/01 ,  G06K9/00

Abstract: Machine learning systems and methods that learn glare, and thus determine gaze direction in a manner more resilient to the effects of glare on input images. The machine learning systems have an isolated representation of glare, e.g., information on the locations of glare points in an image, as an explicit input, in addition to the image itself In this manner, the machine learning systems explicitly consider glare while making a determination of gaze direction, thus producing more accurate results for images containing glare.