公开(公告)号：US11818474B1

公开(公告)日：2023-11-14

申请号：US17895875

申请日：2022-08-25

Applicant: Meta Platforms Technologies, LLC

Inventor： Alex Locher ,  Naveen Makineni ,  Oskar Linde ,  John Enders Robertson ,  Anthony Aslan Tenggoro

IPC: H04N25/133 ,  H04N23/84 ,  H04N25/13 ,  G06T3/40 ,  G06T19/00 ,  H04N13/388

CPC classification number: H04N25/133 ,  G06T3/4015 ,  H04N23/843 ,  H04N25/134 ,  G06T19/006 ,  H04N13/388

Abstract: In one embodiment, a computing system may receive sensor data from an image sensor having a pixel array including color pixel sensors and panchromatic pixel sensors in a first pattern. Each of the color pixel sensors is associated with one of several color channels. The computing system may generate, based on the sensor data, a filtered monochrome image including monochrome values corresponding to the pixel array of the image sensor. The computing system may generate a filtered color image having a second pattern of color channels. A first pixel of a particular color channel at a first pixel location in the filtered color image is determined based on the monochrome value corresponding to the first pixel location in the filtered monochrome image, the sensor data measured by a color pixel sensor at a second pixel location, and the monochrome value at the second pixel location in the filtered monochrome image.

公开(公告)号：US20240153223A1

公开(公告)日：2024-05-09

申请号：US18504944

申请日：2023-11-08

Applicant: Meta Platforms Technologies, LLC

Inventor： Andrey Tovchigrechko ,  Fabian Langguth ,  Alex Locher ,  Britta Hummel ,  Paul Timothy Furgale ,  Ricardo da Silveira Cabral ,  Sebastian Sztuk

IPC: G06T19/00 ,  G06T5/20 ,  G06V10/26

CPC classification number: G06T19/006 ,  G06T5/20 ,  G06V10/26

Abstract: In particular embodiments, a computing system may capture access a set of depth measurements and an image of a scene generated using one or more sensors of an artificial reality device. The system may generate, based on the image, a plurality of segmentation masks respectively associated with a plurality of object types. The system may segment, using the plurality of segmentation masks, the set of depth measurements into subsets of depth measurements respectively associated with the plurality of object types. The system may determine, for each object type, a three-dimensional (3D) model that best fits the subset of depth measurements corresponding to the object type. The system may refine, using 3D models determined for the plurality of object types, the subsets of depth measurements respectively associated with the plurality of object types and use refined depth measurements for mixed reality rendering.

公开(公告)号：US20250157130A1

公开(公告)日：2025-05-15

申请号：US18418130

申请日：2024-01-19

Applicant: Meta Platforms Technologies, LLC

Inventor： Andrey Tovchigrechko ,  Alexander Sorkine Hornung ,  Sebastian Sztuk ,  Remo Ziegler ,  Oskar Linde ,  Ricardo da Silveira Cabral ,  Alex Locher ,  Anirud Thyagharajan ,  Artem Filatov ,  Fabian Langguth ,  Ngoc Minh Dang ,  Salah Eddine Nouri ,  Paul Timothy Furgale

IPC: G06T15/10 ,  G06T5/77 ,  G06T7/194 ,  G06T7/50 ,  G06T19/00

Abstract: A device may access an input image of a real-world scene captured by a camera from a camera viewpoint. The device may render, from the camera viewpoint, an inpainting image of a background of the real-world scene based on a 3D reconstruction model of the background of the real-world scene, wherein the 3D reconstruction model is generated using previously-captured images and previously-generated depth estimates. The device may generate a depth estimate of the real-world scene and identify, based on the depth estimate, a first set of pixel locations and a second set of pixel locations in a passthrough image to be rendered. The device may render, from a viewpoint of an eye of a user, the passthrough image based on the input image, the inpainting image, and the depth estimate. First pixel values for the first set of pixel locations in the passthrough image are sampled from the input image, and second pixel values for the second set of pixel locations in the passthrough image are sampled from the inpainting image.

公开(公告)号：US11830148B2

公开(公告)日：2023-11-28

申请号：US16944045

申请日：2020-07-30

Applicant: META PLATFORMS TECHNOLOGIES, LLC

Inventor： James Allan Booth ,  Gaurav Chaurasia ,  Alexandru-Eugen Ichim ,  Alex Locher ,  Gioacchino Noris ,  Alexander Sorkine Hornung ,  Manuel Werlberger

IPC: G06T19/00 ,  G06T15/04 ,  G06T17/20

CPC classification number: G06T19/006 ,  G06T15/04 ,  G06T17/20

Abstract: A mixed reality (MR) simulation system includes a console and a head mounted device (HMD). The MR system captures stereoscopic images from a real-world environment using outward-facing stereoscopic cameras mounted to the HMD. The MR system preprocesses the stereoscopic images to maximize contrast and then extracts a set of features from those images, including edges or corners, among others. For each feature, the MR system generates one or more two-dimensional (2D) polylines. Then, the MR system triangulates between 2D polylines found in right side images and corresponding 2D polylines found in left side images to generate a set of 3D polylines. The MR system interpolates between 3D vertices included in the 3D polylines or extrapolates additional 3D vertices, thereby generating a geometric reconstruction of the real-world environment. The MR system may map textures derived from the real-world environment onto the geometric representation faster than the geometric reconstruction is updated.