-
公开(公告)号：US20220237843A1
公开(公告)日：2022-07-28
申请号：US17580486
申请日：2022-01-20
Applicant: Facebook Technologies, LLC
Inventor： Jason Saragih , Stephen Anthony Lombardi , Shunsuke Saito , Tomas Simon Kreuz , Shih-En Wei , Kevyn Alex Anthony McPhail , Yaser Sheikh , Sai Bi
Abstract: A method for providing a relightable avatar of a subject to a virtual reality application is provided. The method includes retrieving multiple images including multiple views of a subject and generating an expression-dependent texture map and a view-dependent texture map for the subject, based on the images. The method also includes generating, based on the expression-dependent texture map and the view-dependent texture map, a view of the subject illuminated by a light source selected from an environment in an immersive reality application, and providing the view of the subject to an immersive reality application running in a client device. A non-transitory, computer-readable medium storing instructions and a system that executes the instructions to perform the above method are also provided.
-
公开(公告)号：US20220198731A1
公开(公告)日：2022-06-23
申请号：US17556367
申请日：2021-12-20
Applicant: Facebook Technologies, LLC
Inventor： Stephen Anthony Lombardi , Jason Saragih , Tomas Simon Kreuz , Shunsuke Saito , Michael Zollhoefer , Amit Raj , James Henry Hays
Abstract: A method of forming a pixel-aligned volumetric avatar includes receiving multiple two-dimensional images having at least two or more fields of view of a subject. The method also includes extracting multiple image features from the two-dimensional images using a set of learnable weights, projecting the image features along a direction between a three-dimensional model of the subject and a selected observation point for a viewer, and providing, to the viewer, an image of the three-dimensional model of the subject. A system and a non-transitory, computer readable medium storing instructions to perform the above method, are also provided.
-
公开(公告)号：US20220207831A1
公开(公告)日：2022-06-30
申请号：US17556429
申请日：2021-12-20
Applicant: Facebook Technologies, LLC
Inventor： Jason Saragih , Shih-En Wei , Tomas Simon Kreuz , Kris Makoto Kitani , Ye Yuan
Abstract: A method for simulating a solid body animation of a subject includes retrieving a first frame that includes a body image of a subject. The method also includes selecting, from the first frame, multiple key points within the body image of the subject that define a hull of a body part and multiple joint points that define a joint between two body parts, identifying a geometry, a speed, and a mass of the body part to include in a dynamic model of the subject, based on the key points and the joint points, determining, based on the dynamic model of the subject, a pose of the subject in a second frame after the first frame in a video stream, and providing the video stream to an immersive reality application running on a client device.
-
公开(公告)号：US20220201273A1
公开(公告)日：2022-06-23
申请号：US17555037
申请日：2021-12-17
Applicant: Facebook Technologies, LLC
Inventor： Nathan Matsuda , Brian Wheelwright , Joel Hegland , Stephen Anthony Lombardi , Jason Saragih , Tomas Simon Kreuz , Shunsuke Saito , Michael Zollhoefer , Amit Raj , James Henry Hays
IPC: H04N13/344 , G06T7/80 , G06T15/00 , H04N13/117 , H04N13/239 , H04N5/33 , H04N13/383 , G02B30/10
Abstract: A device for providing a reverse pass-through view of a user of a headset display to an onlooker includes an eyepiece comprising an optical surface configured to provide an image to a user on a first side of the optical surface. The device also includes a first camera configured to collect an image of a portion of a face of the user reflected from the optical surface in a first field of view, a display adjacent to the optical surface and configured to project forward an image of the face of the user, and a screen configured to receive light from the display and provide the image of the face of the user to an onlooker.
-
公开(公告)号：US11182947B1
公开(公告)日：2021-11-23
申请号：US16852226
申请日：2020-04-17
Applicant: Facebook Technologies, LLC
Inventor： Chenglei Wu , Jason Saragih , Tomas Simon Kreuz , Takaaki Shiratori
Abstract: In one embodiment, a system may access a codec that encodes an appearance associated with a subject and comprise codec portions that respectively correspond to body parts of the subject. The system may generate a training codec that comprises a first subset of the codec portions (a first set of body parts) and a modified second subset of the codec portions (muted body parts). The system may decode the training codec using a machine-learning model to generate a mesh of the subject. The system may transform the mesh of the subject based on a predetermined pose. The system may update the machine-learning model based on a comparison between the transformed mesh and a target mesh of the subject having the predetermined pose. The system in the present application can train a machine-learning model to render an avatar with a pose using uncorrelated codec portions corresponding to different body parts.
-
Patent Agency Ranking
公开(公告)号：US11087521B1
公开(公告)日：2021-08-10
申请号：US16775953
申请日：2020-01-29
Applicant: Facebook Technologies, LLC
Inventor： Stephen Anthony Lombardi , Jason Saragih , Yaser Sheikh , Takaaki Shiratori , Shoou-I Yu , Tomas Simon Kreuz , Chenglei Wu
Abstract: The disclosed computer system may include an input module, an autoencoder, and a rendering module. The input module may receive geometry information and images of a subject. The geometry information may be indicative of variation in geometry of the subject over time. Each image may be associated with a respective viewpoint and may include a view-dependent texture map of the subject. The autoencoder may jointly encode texture information and the geometry information to provide a latent vector. The autoencoder may infer, using the latent vector, an inferred geometry and an inferred view-dependent texture of the subject for a predicted viewpoint. The rendering module may be configured to render a reconstructed image of the subject for the predicted viewpoint using the inferred geometry and the inferred view-dependent texture. Various other systems and methods are also disclosed.
-
公开(公告)号：US11062502B2
公开(公告)日：2021-07-13
申请号：US16379615
申请日：2019-04-09
Applicant: Facebook Technologies, LLC
Abstract: In one embodiment, a method includes accessing a number of pictures of an object, constructing a modeling volume for three-dimensional modeling of the object by processing the number of pictures using a machine-learning framework, where the modeling volume is associated with a number of color and opacity information that are associated with a number of regions in the modeling volume, and rendering an image of the object from a view-point using the modeling volume, where each pixel of the image is rendered by projecting a virtual ray from the view-point and through the modeling volume, determining one or more of the number of regions in the modeling volume intersected by the virtual ray, and determining a color and an opacity of the pixel based on an accumulation of the color and opacity information associated with the one or more of the number of regions intersected by the virtual ray.
-
公开(公告)号：US20200327718A1
公开(公告)日：2020-10-15
申请号：US16379615
申请日：2019-04-09
Applicant: Facebook Technologies, LLC
Abstract: In one embodiment, a method includes accessing a number of pictures of an object, constructing a modeling volume for three-dimensional modeling of the object by processing the number of pictures using a machine-learning framework, where the modeling volume is associated with a number of color and opacity information that are associated with a number of regions in the modeling volume, and rendering an image of the object from a view-point using the modeling volume, where each pixel of the image is rendered by projecting a virtual ray from the view-point and through the modeling volume, determining one or more of the number of regions in the modeling volume intersected by the virtual ray, and determining a color and an opacity of the pixel based on an accumulation of the color and opacity information associated with the one or more of the number of regions intersected by the virtual ray.
-
公开(公告)号：US11423616B1
公开(公告)日：2022-08-23
申请号：US16833360
申请日：2020-03-27
Applicant: Facebook Technologies, LLC
Inventor： Tomas Simon Kreuz , Jason Saragih , Stephen Anthony Lombardi , Shugao Ma , Gabriel Bailowitz Schwartz
Abstract: In one embodiment, a system may access an input image of an object captured by cameras, and the input image depicts appearance information associated with an object. The system may generate a first mesh of the object based on features identified from the input image of the object. The system may generate, by processing the first mesh using a machine-learning model, a position map that defines a contour of the object. Each pixel in the position map corresponds to a three-dimensional coordinate. The system may further generate a second mesh based on the position map, wherein the second mesh has a higher resolution than the first mesh. The system may render an output image of the object based on the second mesh. The system disclosed in the present application can render a dense mesh which has a higher resolution to provide details which cannot be compensated by texture information.
-
公开(公告)号：US20220245910A1
公开(公告)日：2022-08-04
申请号：US17554992
申请日：2021-12-17
Applicant: Facebook Technologies, LLC
Inventor： Stephen Anthony Lombardi , Tomas Simon Kreuz , Jason Saragih , Gabriel Bailowitz Schwartz , Michael Zollhoefer , Yaser Sheikh
Abstract: A method for training a real-time, modeling for animating an avatar for a subject is provided. The method includes collecting multiple images of a subject. The method also includes selecting a plurality of vertex positions in a guide mesh, indicative of a volumetric primitive enveloping the subject, determining a geometric attribute for the volumetric primitive including a position, a rotation, and a scale factor of the volumetric primitive, determining a payload attribute for each of the volumetric primitive, the payload attribute including a color value and an opacity value for each voxel in a voxel grid defining the volumetric primitive, determining a loss factor for each point in the volumetric primitive based on the geometric attribute, the payload attribute and a ground truth value, and updating a three-dimensional model for the subject. A system and a non-transitory, computer-readable medium storing instructions to perform the above method are also provided.
-
-
-
-
-
-
-
-
-
Search Results
11
records
(took 0.04 seconds)
