公开(公告)号：US20250157130A1

公开(公告)日：2025-05-15

申请号：US18418130

申请日：2024-01-19

Applicant: Meta Platforms Technologies, LLC

Inventor： Andrey Tovchigrechko ,  Alexander Sorkine Hornung ,  Sebastian Sztuk ,  Remo Ziegler ,  Oskar Linde ,  Ricardo da Silveira Cabral ,  Alex Locher ,  Anirud Thyagharajan ,  Artem Filatov ,  Fabian Langguth ,  Ngoc Minh Dang ,  Salah Eddine Nouri ,  Paul Timothy Furgale

IPC: G06T15/10 ,  G06T5/77 ,  G06T7/194 ,  G06T7/50 ,  G06T19/00

Abstract: A device may access an input image of a real-world scene captured by a camera from a camera viewpoint. The device may render, from the camera viewpoint, an inpainting image of a background of the real-world scene based on a 3D reconstruction model of the background of the real-world scene, wherein the 3D reconstruction model is generated using previously-captured images and previously-generated depth estimates. The device may generate a depth estimate of the real-world scene and identify, based on the depth estimate, a first set of pixel locations and a second set of pixel locations in a passthrough image to be rendered. The device may render, from a viewpoint of an eye of a user, the passthrough image based on the input image, the inpainting image, and the depth estimate. First pixel values for the first set of pixel locations in the passthrough image are sampled from the input image, and second pixel values for the second set of pixel locations in the passthrough image are sampled from the inpainting image.

公开(公告)号：US20240338073A1

公开(公告)日：2024-10-10

申请号：US18602034

申请日：2024-03-12

Applicant: Meta Platforms Technologies, LLC

Inventor： Ilya Brailovskiy ,  Sebastian Sztuk ,  Oskar Linde ,  Daniel Maskit ,  Devraj Gupta ,  Zheng Liang ,  Rui Xiao ,  Gary King

IPC: G06F3/01 ,  G06T3/4053 ,  G06T5/70 ,  G06T5/77 ,  G06T7/50 ,  G06T19/00

CPC classification number: G06F3/013 ,  G06T3/4053 ,  G06T5/70 ,  G06T5/77 ,  G06T7/50 ,  G06T19/00

Abstract: In one embodiment, a computing system may determine a gaze of a user of the computing system. The computing system may generate a foveated map based on the gaze to determine a sensor readout for an image sensor of the computing system. The foveated map may include several foveal regions. The computing system may determine the sensor readout including several zones corresponding to the image sensor based on the several foveal regions. Each of the several zones may indicate a readout resolution for an area of the image sensor for the respective zone. The computing system may capture a first image using the image sensor. The computing system may generate a modified first image based on the captured first image and the sensor readout.

公开(公告)号：US11818474B1

公开(公告)日：2023-11-14

申请号：US17895875

申请日：2022-08-25

Applicant: Meta Platforms Technologies, LLC

Inventor： Alex Locher ,  Naveen Makineni ,  Oskar Linde ,  John Enders Robertson ,  Anthony Aslan Tenggoro

IPC: H04N25/133 ,  H04N23/84 ,  H04N25/13 ,  G06T3/40 ,  G06T19/00 ,  H04N13/388

CPC classification number: H04N25/133 ,  G06T3/4015 ,  H04N23/843 ,  H04N25/134 ,  G06T19/006 ,  H04N13/388

Abstract: In one embodiment, a computing system may receive sensor data from an image sensor having a pixel array including color pixel sensors and panchromatic pixel sensors in a first pattern. Each of the color pixel sensors is associated with one of several color channels. The computing system may generate, based on the sensor data, a filtered monochrome image including monochrome values corresponding to the pixel array of the image sensor. The computing system may generate a filtered color image having a second pattern of color channels. A first pixel of a particular color channel at a first pixel location in the filtered color image is determined based on the monochrome value corresponding to the first pixel location in the filtered monochrome image, the sensor data measured by a color pixel sensor at a second pixel location, and the monochrome value at the second pixel location in the filtered monochrome image.

公开(公告)号：US20240153182A1

公开(公告)日：2024-05-09

申请号：US18053276

申请日：2022-11-07

Applicant: Meta Platforms Technologies, LLC

Inventor： John Robert Volk ,  Min Tao ,  Oskar Linde ,  Adam Alan Mudrick ,  Alexander Randon Cope

IPC: G06T13/40 ,  G06F3/01 ,  G06T17/10

CPC classification number: G06T13/40 ,  G06F3/012 ,  G06F3/013 ,  G06F3/017 ,  G06T17/10

Abstract: A method for updating a subject avatar with a facial expression collected in real time. The method includes receiving, from a sensor on a facial interface of a headset, a signal indicative of a movement in a facial muscle from a user of the headset, determining a facial expression of the user with the signal from the sensor in the facial interface of the headset, based on a machine learning algorithm trained to associate the facial expression to the movement of the facial muscle, adjusting a subject avatar for the user of the headset based on the facial expression, and providing the subject avatar to an immersive reality application hosted by a remote server. A headset, a computer-readable medium storing instructions which, when executed by a processor cause the headset to perform the above method, and the processor, are also provided.

公开(公告)号：US20240119568A1

公开(公告)日：2024-04-11

申请号：US18484193

申请日：2023-10-10

Applicant: Meta Platforms Technologies, LLC

Inventor： Andrey Tovchigrechko ,  Fabian Langguth ,  Alexander Sorkine Hornung ,  Oskar Linde ,  Christian Forster

IPC: G06T5/00 ,  G06T7/10 ,  G06T7/285 ,  G06T17/20 ,  G06T19/20 ,  G06V20/20

CPC classification number: G06T5/002 ,  G06T5/005 ,  G06T7/10 ,  G06T7/285 ,  G06T17/20 ,  G06T19/20 ,  G06V20/20 ,  G06T2207/10012 ,  G06T2207/10028 ,  G06T2207/20021 ,  G06T2210/44 ,  G06T2219/2021

Abstract: A processor accesses a depth map and a first image of a scene generated using one or more sensors of an artificial reality device. The processor generates, based on the first image, segmentation masks respectively associated with a plurality of object types. The segmentation masks segment the depth map into a plurality of segmented depth maps respectively associated with the object types. The processor generates meshes using, respectively, the segmented depth maps. For each eye of the user, the processor captures a second image and generates, based on the second image, segmentation information. The processor warps the plurality of meshes to generate warped meshes for the eye, and then generates an eye-specific mesh for the eye by compositing the warped meshes according to the segmentation information. The processor renders an output image for the eye using the second image and the eye-specific mesh.