公开(公告)号：US20230092248A1

公开(公告)日：2023-03-23

申请号：US17805828

申请日：2022-06-07

Applicant: Samsung Electronics Co., Ltd.

Inventor： Yingen Xiong ,  Christopher A. Peri

IPC: G06T7/11 ,  G06V10/25 ,  G06V20/20 ,  G06T7/50 ,  G06V10/82 ,  G06V10/764 ,  G06V10/32

Abstract: A method includes obtaining, from an image sensor, image data of a real-world scene; obtaining, from a depth sensor, sparse depth data of the real-world scene; and passing the image data to a first neural network to obtain one or more object regions of interest (ROIs) and one or more feature map ROIs. Each object ROI includes at least one detected object. The method also includes passing the image data and sparse depth data to a second neural network to obtain one or more dense depth map ROIs; aligning the one or more object ROIs, one or more feature map ROIs, and one or more dense depth map ROIs; and passing the aligned ROIs to a fully convolutional network to obtain a segmentation of the real-world scene. The segmentation contains one or more pixelwise predictions of one or more objects in the real-world scene.

公开(公告)号：US11468587B2

公开(公告)日：2022-10-11

申请号：US17093519

申请日：2020-11-09

Applicant: Samsung Electronics Co., Ltd.

Inventor： Christopher A. Peri ,  Yingen Xiong

IPC: G06T7/593 ,  G06T7/73 ,  G06T3/40 ,  G06T5/50

Abstract: A method for reconstructing a downsampled depth map includes receiving, at an electronic device, image data to be presented on a display of the electronic device at a first resolution, wherein the image data includes a color image and the downsampled depth map associated with the color image. The method further includes generating a high resolution depth map by calculating, for each point making up the first resolution, a depth value based on a normalized pose difference across a neighborhood of points for the point, a normalized color texture difference across the neighborhood of points for the point, and a normalized spatial difference across the neighborhood of points. Still further, the method includes outputting, on the display, a reprojected image at the first resolution based on the color image and the high resolution depth map. The downsampled depth map is at a resolution less than the first resolution.

公开(公告)号：US20210358158A1

公开(公告)日：2021-11-18

申请号：US17093519

申请日：2020-11-09

Applicant: Samsung Electronics Co., Ltd.

Inventor： Christopher A. Peri ,  Yingen Xiong

IPC: G06T7/593 ,  G06T7/73 ,  G06T5/50 ,  G06T3/40

Abstract: A method for reconstructing a downsampled depth map includes receiving, at an electronic device, image data to be presented on a display of the electronic device at a first resolution, wherein the image data includes a color image and the downsampled depth map associated with the color image. The method further includes generating a high resolution depth map by calculating, for each point making up the first resolution, a depth value based on a normalized pose difference across a neighborhood of points for the point, a normalized color texture difference across the neighborhood of points for the point, and a normalized spatial difference across the neighborhood of points. Still further, the method includes outputting, on the display, a reprojected image at the first resolution based on the color image and the high resolution depth map. The downsampled depth map is at a resolution less than the first resolution.

公开(公告)号：US20250148701A1

公开(公告)日：2025-05-08

申请号：US18759361

申请日：2024-06-28

Applicant: Samsung Electronics Co., Ltd.

Inventor： Yingen Xiong

IPC: G06T15/40 ,  G06T7/13 ,  G06T7/55 ,  G06T19/00

Abstract: A method includes obtaining image frames of a scene captured using one or more imaging sensors of a video see-through (VST) extended reality (XR) device and depth data associated with the scene. The image frames capture a moving object and static scene contents, and the moving object includes a portion of a body of a user. The method also includes generating masks associated with the moving object using a machine learning model trained to separate pixels corresponding to human skin from other portions of the scene. The method further includes reconstructing images of the moving object and images of the static scene contents. In addition, the method includes combining the images of the moving object, the images of the static scene contents, and one or more virtual features to generate combined images and rendering the combined images for presentation on at least one display of the VST XR device.

公开(公告)号：US12062145B2

公开(公告)日：2024-08-13

申请号：US17811012

申请日：2022-07-06

Applicant: Samsung Electronics Co., Ltd.

Inventor： Yingen Xiong ,  Christopher A. Peri

IPC: G06T19/00 ,  G01B11/22 ,  G01C21/16 ,  G06T7/50 ,  G06T7/70 ,  G06T17/20 ,  G06V10/25 ,  G06V10/26 ,  G06V10/77 ,  G06V20/20

CPC classification number: G06T19/006 ,  G01B11/22 ,  G01C21/1656 ,  G06T7/50 ,  G06T7/70 ,  G06T17/20 ,  G06V10/25 ,  G06V10/26 ,  G06V10/7715 ,  G06V20/20 ,  G06T2207/30244

Abstract: A method includes receiving depth data of a real-world scene from a depth sensor, receiving image data of the scene from an image sensor, receiving movement data of the depth and image sensors from an IMU, and determining an initial 6DOF pose of an apparatus based on the depth data, image data, and/or movement data. The method also includes passing the 6DOF pose to a back end to obtain an optimized pose and generating, based on the optimized pose, image data, and depth data, a three-dimensional reconstruction of the scene. The reconstruction includes a dense depth map, a dense surface mesh, and/or one or more semantically segmented objects. The method further includes passing the reconstruction to a front end and rendering, at the front end, an XR frame. The XR frame includes a three-dimensional XR object projected on one or more surfaces of the scene.

公开(公告)号：US11900845B2

公开(公告)日：2024-02-13

申请号：US17696729

申请日：2022-03-16

Applicant: Samsung Electronics Co., Ltd.

Inventor： Yingen Xiong ,  Christopher A. Peri

IPC: G09G3/20

CPC classification number: G09G3/2003 ,  G09G2320/0242 ,  G09G2320/0285

Abstract: A system and method for display distortion calibration are configured to capture distortion with image patterns and calibrate distortion with ray tracing for an optical pipeline with lenses. The system includes an image sensor and a processor to perform the method for display distortion calibration. The method includes generating an image pattern to encode display image pixels by encoding display distortion associated with a plurality of image patterns. The method also includes determining a distortion of the image pattern resulting from a lens on a head-mounted display (HMD) and decoding the distorted image patterns to obtain distortion of pixels on a display. A lookup table is created of angular distortion of all the pixels on the display. The method further includes providing a compensation factor for the distortion by creating distortion correction based on the lookup table of angular distortion.

公开(公告)号：US20240046575A1

公开(公告)日：2024-02-08

申请号：US18135985

申请日：2023-04-18

Applicant: Samsung Electronics Co., Ltd.

Inventor： Yingen Xiong

IPC: G06T19/00 ,  H04N13/239 ,  G06T7/80 ,  G06T7/593 ,  H04N13/111

CPC classification number: G06T19/006 ,  H04N13/239 ,  G06T7/85 ,  G06T7/593 ,  H04N13/111 ,  H04N2013/0081

Abstract: In one embodiment, a method includes capturing, using a pair of cameras of a video see-through AR system, stereo images of a predetermined calibration object having a predetermined pose and obtaining, by a position sensor, position data for each of the pair of cameras at a time when the stereo images are captured. The method further includes generating a 3D reconstruction of the calibration object. The method further includes performing a registration process by generating, using parameters of each of a pair of stereo virtual cameras, stereo virtual views that include the predetermined calibration object and determining, for each of the stereo virtual views, one or more differences between the predetermined calibration object in the virtual images and a virtual rendering of the calibration object, and then using those differences to either adjust the virtual-camera parameters and repeat the registration process or to store the parameters for the system.

公开(公告)号：US20230410414A1

公开(公告)日：2023-12-21

申请号：US18052827

申请日：2022-11-04

Applicant: Samsung Electronics Co., Ltd.

Inventor： Yingen Xiong ,  Christopher A. Peri

IPC: G06T15/20 ,  G06T7/55 ,  G06T5/00 ,  G06F3/01 ,  G06T19/00

CPC classification number: G06T15/20 ,  G06T7/55 ,  G06T2207/10016 ,  G06F3/013 ,  G06T19/006 ,  G06T5/005

Abstract: A method of video transformation for a video see-through (VST) augmented reality (AR) device includes obtaining video frames from multiple cameras associated with the VST AR device, where each video frame is associated with position data. The method also includes generating camera viewpoint depth maps associated with the video frames based on the video frames and the position data. The method further includes performing depth re-projection to transform the video frames from camera viewpoints to rendering viewpoints using the camera viewpoint depth maps. The method also includes performing hole filling of one or more holes created in one or more occlusion areas of at least one of the transformed video frames during the depth re-projection to generate at least one hole-filled video frame. In addition, the method includes displaying the transformed video frames including the at least one hole-filled video frame on multiple displays associated with the VST AR device.

公开(公告)号：US20220406013A1

公开(公告)日：2022-12-22

申请号：US17500817

申请日：2021-10-13

Applicant: SAMSUNG ELECTRONICS CO., LTD.

Inventor： Yingen Xiong ,  Christopher A. Peri

IPC: G06T17/20 ,  G06T3/40 ,  G06T19/00 ,  G06N20/00 ,  G06T7/55 ,  G06T19/20

Abstract: Generating a 3D scene reconstruction using depth fusion can include creating a high-resolution sparse depth map by mapping sensor depths from a low-resolution depth map to points corresponding to pixels of a high-resolution color image of a scene. The high-resolution sparse depth map can have the same resolution as the high-resolution color image. A fused sparse depth map can be produced by combining the high-resolution sparse depth map with sparse depths reconstructed from the high-resolution color image. The high-resolution dense depth map can be generated based on fused sparse depths of the fused sparse depth map.

公开(公告)号：US11481871B2

公开(公告)日：2022-10-25

申请号：US17402005

申请日：2021-08-13

Applicant: SAMSUNG ELECTRONICS CO., LTD.

Inventor： Yingen Xiong ,  Christopher A. Peri

IPC: G06T3/00 ,  G06T7/50 ,  G06T7/13 ,  G06F3/01

Abstract: Updating an image during real-time rendering of images by a display device can include determining a depth for each pixel of a color frame received from a source device and corresponding to the image. Each pixel's depth is determined by image-guided propagation of depths of sparse points extracted from a depth map generated at the source device. With respect to pixels corresponding to an extracted sparse depth point, image-guided depth propagation can include retaining the depth of the corresponding sparse depth point unchanged from the source depth map. With respect to each pixel corresponding to a non-sparse depth point, image-guided depth propagation can include propagating to the corresponding non-sparse depth point a depth of a sparse depth point lying within a neighborhood of the non-sparse depth point. Pixel coordinates of the color frame can be transformed for generating a space-warped rendering of the image.