公开(公告)号：US20210241462A1

公开(公告)日：2021-08-05

申请号：US17219555

申请日：2021-03-31

Applicant: SHANGHAITECH UNIVERSITY

Inventor： Ziran XING ,  Zhiru SHI ,  Yi MA ,  Jingyi YU

IPC: G06T7/11 ,  G06T7/50 ,  G06T7/187 ,  G06T7/162

Abstract: According to some embodiments, an imaging processing method for extracting a plurality of planar surfaces from a depth map includes computing a depth change indication map (DCI) from a depth map in accordance with a smoothness threshold. The imaging processing method further includes recursively extracting a plurality of planar region from the depth map, wherein the size of each planar region is dynamically adjusted according to the DCI. The imaging processing method further includes clustering the extracted planar regions into a plurality of groups in accordance with a distance function; and growing each group to generate pixel-wise segmentation results and inlier points statistics simultaneously.

公开(公告)号：US20210082096A1

公开(公告)日：2021-03-18

申请号：US17074123

申请日：2020-10-19

Applicant: SHANGHAITECH UNIVERSITY

Inventor： Minye WU ,  Zhiru SHI ,  Jingyi YU

IPC: G06T5/50 ,  G06T7/557 ,  G06T7/593

Abstract: A method of processing light field images for separating a transmitted layer from a reflection layer. The method comprises capturing a plurality of views at a plurality of viewpoints with different polarization angles; obtaining an initial disparity estimation for a first view using SIFT-flow, and warping the first view to a reference view; optimizing an objective function comprising a transmitted layer and a secondary layer using an Augmented Lagrange Multiplier (ALM) with Alternating Direction Minimizing (ADM) strategy; updating the disparity estimation for the first view; repeating the steps of optimizing the objective function and updating the disparity estimation until the change in the objective function between two consecutive iterations is below a threshold; and separating the transmitted layer and the secondary layer using the disparity estimation for the first view.

公开(公告)号：US20200302155A1

公开(公告)日：2020-09-24

申请号：US16894032

申请日：2020-06-05

Applicant: SHANGHAITECH UNIVERSITY

Inventor： Zhiru SHI ,  Minye WU ,  Wengguang MA ,  Jingyi YU

IPC: G06K9/00 ,  H04N13/282 ,  H04N5/225

Abstract: A method of detecting and recognizing faces using a light field camera array is provided. The method includes capturing multi-view color images using the light field camera array; obtaining a depth map; conducting light field rendering using a weight function comprising a depth component and a sematic component, where the weight function assigns a ray in the light field with a weight; and detecting and recognizing a face.

公开(公告)号：US20240290059A1

公开(公告)日：2024-08-29

申请号：US18571748

申请日：2021-07-26

Applicant: SHANGHAITECH UNIVERSITY

Inventor： Jiakai ZHANG ,  Jingyi YU ,  Lan XU

IPC: G06V10/25 ,  G06T3/02 ,  G06T7/50

CPC classification number: G06V10/25 ,  G06T3/02 ,  G06T7/50

Abstract: A computer-implemented method of generating editable free-viewport videos is provided. A plurality of video of a scene from a plurality of views is obtained. The scene comprises includes an environment and one or more dynamic entities. A 3D bounding-box is generated for each dynamic entity in the scene. A computer device encodes a machine learning model including an environment layer and a dynamic entity layer for each dynamic entity in the scene. The environment layer represents a continuous function of space and time of the environment. The dynamic entity layer represents a continuous function of space and time of the dynamic entity. The dynamic entity layer includes a deformation module and a neural radiance module. The deformation module is configured to deform a spatial coordinate in accordance with a timestamp and a trained deformation weight. The neural radiance module is configured to derive a density value and a color.

公开(公告)号：US20240104822A1

公开(公告)日：2024-03-28

申请号：US18531755

申请日：2023-12-07

Applicant: SHANGHAITECH UNIVERSITY

Inventor： Chaolin RAO ,  Minye WU ,  Xin LOU ,  Pingqiang ZHOU ,  Jingyi YU

IPC: G06T15/00 ,  G06T7/90

CPC classification number: G06T15/005 ,  G06T7/90 ,  G06T2207/20084

Abstract: An image rendering system comprising a preprocessing unit coupled to a feature extract unit and a color rendering unit over a data bus. The preprocessing unit generates vector representations of spatial coordinates of sample points along camera rays corresponding to pixels of an image to be rendered. The feature extract unit generates a feature map of the image based on the vector representations, color and intensity values of the sample point through a first machine learning model. The color rendering unit renders the image based on the feature map through a second machine learning model. The first machine learning model is different from the second machine learning model.

公开(公告)号：US20230360372A1

公开(公告)日：2023-11-09

申请号：US18223575

申请日：2023-07-19

Applicant: SHANGHAITECH UNIVERSITY

Inventor： Fuqiang ZHAO ,  Minye WU ,  Lan XU ,  Jingyi YU

IPC: G06V10/774 ,  G06V10/82 ,  G06T7/50 ,  G06T7/80 ,  G06V10/74 ,  G06V20/64

CPC classification number: G06V10/774 ,  G06V10/82 ,  G06T7/50 ,  G06T7/80 ,  G06V10/761 ,  G06V20/64 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/10028 ,  G06V40/172

Abstract: Systems, methods, and non-transitory computer-readable media are configured to obtain a set of content items to train a neural radiance field-based (NeRF-based) machine learning model for object recognition. Depth maps of objects depicted in the set of content items can be determined. A first set of training data comprising reconstructed content items depicting only the objects can be generated based on the depth maps. A second set of training data comprising one or more optimal training paths associated with the set of content items can be generated based on the depth maps. The one or more optimal training paths are generated based at least in part on a dissimilarity matrix associated with the set of content items. The NeRF-based machine learning model can be trained based on the first set of training data and the second set of training data.

公开(公告)号：US20230027234A1

公开(公告)日：2023-01-26

申请号：US17951405

申请日：2022-09-23

Applicant: Shanghaitech University

Inventor： Minye WU ,  Jingyi YU

IPC: G06T17/20 ,  G06T7/593 ,  H04N13/207 ,  G06K9/62 ,  G06T15/04 ,  G06T15/20

Abstract: An image-based method of modeling and rendering a three-dimensional model of an object is provided. The method comprises: obtaining a three-dimensional point cloud at each frame of a synchronized, multi-view video of an object, wherein the video comprises a plurality of frames; extracting a feature descriptor for each point in the point cloud for the plurality of frames without storing the feature descriptor for each frame; producing a two-dimensional feature map for a target camera; and using an anti-aliased convolutional neural network to decode the feature map into an image and a foreground mask.

公开(公告)号：US20190028707A1

公开(公告)日：2019-01-24

申请号：US16069185

申请日：2016-01-18

Applicant: SHANGHAITECH UNIVERSITY

Inventor： Jingyi YU ,  Yi MA

IPC: H04N19/126 ,  H04N13/106 ,  H04N19/523 ,  H04N19/159 ,  G06T3/40 ,  H04N19/176

Abstract: A method of compressing a stereoscopic video including a left view frame and a right view frame is provided, the method including: determining a texture saliency value for a first block in the left view frame by intra prediction (1101); determining a motion saliency value for the first block by motion estimation (1102); determining a disparity saliency value between the first block and a corresponding second block in the right view frame (1103); determining a quantization parameter based on the disparity saliency value, the texture saliency value, and the motion saliency value (1104); and performing quantization of the first block in accordance with the quantization parameter (1105).

公开(公告)号：US20240371078A1

公开(公告)日：2024-11-07

申请号：US18574044

申请日：2022-01-11

Applicant: SHANGHAITECH UNIVERSITY

Inventor： Jingyi YU ,  Yuyao ZHANG ,  Xin LOU ,  Qing WU ,  Chaolin RAO ,  Jiawen YANG

IPC: G06T15/08 ,  G06T9/00 ,  G06T19/20

Abstract: An image rendering system for rendering two-dimensional images in real-time. The image rendering system can receive an implicit representation model of a three-dimensional image. The image rendering system can construct, based on voxel coordinates, a three-dimensional image based on the implicit representation model. The image rendering system can rotate the three-dimensional image to an orientation in a computing space based on a user input. The image rendering system can generate a two-dimensional image based on the rotated three-dimensional image.

公开(公告)号：US20240273672A1

公开(公告)日：2024-08-15

申请号：US18568316

申请日：2021-07-12

Applicant: SHANGHAITECH UNIVERSITY

Inventor： Jingyi YU ,  Yuyao ZHANG ,  Lan XU ,  Yuwei LI ,  Qing WU

IPC: G06T3/4046 ,  G06T3/4053 ,  G06T5/10 ,  G06T5/50 ,  G06T5/60

CPC classification number: G06T3/4046 ,  G06T3/4053 ,  G06T5/10 ,  G06T5/50 ,  G06T5/60 ,  G06T2207/10061 ,  G06T2207/10081 ,  G06T2207/10088 ,  G06T2207/20048 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/30016

Abstract: Described herein are methods and non-transitory computer-readable media configured to obtain a plurality of images from a plurality of image scanning orientations for an object. A rigid registration is performed to the plurality of images to obtain a transformation matrix to normalize the plurality of images from their respective image spaces to a normalized image space. Each normalized image comprises a plurality of voxels. A machine learning model comprising an implicit representation of a high-resolution image is trained using the normalized images, wherein the high-resolution image comprises more voxels than the voxels in the normalized images. The high-resolution image is generated based on the trained machine learning model. The plurality of images are a plurality of anisotropic 2D images, while the high resolution image can be a 2D or 3D high resolution image.