公开(公告)号：US20240212262A1

公开(公告)日：2024-06-27

申请号：US18601661

申请日：2024-03-11

Applicant: Meta Platforms Technologies, LLC

Inventor： Brian Keith CABRAL ,  Albert PARRA POZO

IPC: G06T15/20 ,  G06F3/16 ,  H04N7/15

CPC classification number: G06T15/20 ,  G06F3/165 ,  H04N7/157

Abstract: A 3D conversation system can facilitate 3D conversations in an augmented reality environment, allowing conversation participants to appear as if they are face-to-face. The 3D conversation system can accomplish this with a pipeline of data processing stages, which can include calibrate, capture, tag and filter, compress, decompress, reconstruct, render, and display stages. Generally, the pipeline can capture images of the sending user, create intermediate representations, transform the representations to convert from the orientation the images were taken from to a viewpoint of the receiving user, and output images of the sending user, from the viewpoint of the receiving user, in synchronization with audio captured from the sending user. Such a 3D conversation can take place between two or more sender/receiving systems and, in some implementations can be mediated by one or more server systems. In various configurations, stages of the pipeline can be customized based on a conversation context.

公开(公告)号：US20230267675A1

公开(公告)日：2023-08-24

申请号：US18311276

申请日：2023-05-03

Applicant: Meta Platforms Technologies, LLC

Inventor： Brian Keith CABRAL ,  Albert PARRA POZO

IPC: G06T15/20 ,  G06F3/16 ,  H04N7/15

CPC classification number: G06T15/20 ,  G06F3/165 ,  H04N7/157

Abstract: A 3D conversation system can facilitate 3D conversations in an augmented reality environment, allowing conversation participants to appear as if they are face-to-face. The 3D conversation system can accomplish this with a pipeline of data processing stages, which can include calibrate, capture, tag and filter, compress, decompress, reconstruct, render, and display stages. Generally, the pipeline can capture images of the sending user, create intermediate representations, transform the representations to convert from the orientation the images were taken from to a viewpoint of the receiving user, and output images of the sending user, from the viewpoint of the receiving user, in synchronization with audio captured from the sending user. Such a 3D conversation can take place between two or more sender/receiving systems and, in some implementations can be mediated by one or more server systems. In various configurations, stages of the pipeline can be customized based on a conversation context.

公开(公告)号：US20220413434A1

公开(公告)日：2022-12-29

申请号：US17360735

申请日：2021-06-28

Applicant: Meta Platforms Technologies, LLC

Inventor： Albert PARRA POZO ,  Joseph VIRSKUS ,  Ganesh VENKATESH ,  Kai LI ,  Shen-Chi CHEN ,  Amit KUMAR ,  Rakesh RANJAN ,  Brian Keith CABRAL ,  Samuel Alan JOHNSON ,  Wei YE ,  Michael Alexander SNOWER ,  Yash PATEL

IPC: G03H1/00 ,  G03H1/26 ,  H04M3/56 ,  G06T17/20 ,  G06T7/55 ,  G06T7/194 ,  G06N20/00

Abstract: A holographic calling system can capture and encode holographic data at a sender-side of a holographic calling pipeline and decode and present the holographic data as a 3D representation of a sender at a receiver-side of the holographic calling pipeline. The holographic calling pipeline can include stages to capture audio, color images, and depth images; densify the depth images to have a depth value for each pixel while generating parts masks and a body model; use the masks to segment the images into parts needed for hologram generation; convert depth images into a 3D mesh; paint the 3D mesh with color data; perform torso disocclusion; perform face reconstruction; and perform audio synchronization. In various implementations, different of these stages can be performed sender-side or receiver side. The holographic calling pipeline also includes sender-side compression, transmission over a communication channel, and receiver-side decompression and hologram output.

公开(公告)号：US20220413433A1

公开(公告)日：2022-12-29

申请号：US17360693

申请日：2021-06-28

Applicant: Meta Platforms Technologies, LLC

Inventor： Albert PARRA POZO ,  Joseph VIRSKUS ,  Ganesh VENKATESH ,  Kai LI ,  Shen-Chi CHEN ,  Amit KUMAR ,  Rakesh RANJAN ,  Brian Keith CABRAL ,  Samuel Alan JOHNSON ,  Wei YE ,  Michael Alexander SNOWER ,  Yash PATEL

IPC: G03H1/00 ,  G03H1/26 ,  H04M3/56 ,  G06T17/20 ,  G06T7/521 ,  G06T7/55 ,  G06T7/194 ,  G06N20/00 ,  H04N19/597 ,  G06K9/00

Abstract: A holographic calling system can capture and encode holographic data at a sender-side of a holographic calling pipeline and decode and present the holographic data as a 3D representation of a sender at a receiver-side of the holographic calling pipeline. The holographic calling pipeline can include stages to capture audio, color images, and depth images; densify the depth images to have a depth value for each pixel while generating parts masks and a body model; use the masks to segment the images into parts needed for hologram generation; convert depth images into a 3D mesh; paint the 3D mesh with color data; perform torso disocclusion; perform face reconstruction; and perform audio synchronization. In various implementations, different of these stages can be performed sender-side or receiver side. The holographic calling pipeline also includes sender-side compression, transmission over a communication channel, and receiver-side decompression and hologram output.