公开(公告)号：US09269003B2

公开(公告)日：2016-02-23

申请号：US14225157

申请日：2014-03-25

Applicant: QUALCOMM Incorporated

Inventor： Dieter Schmalstieg

IPC: G06K9/00 ,  G06T7/20 ,  G06T19/00 ,  G06T15/00

CPC classification number: G06K9/00624 ,  G06T7/20 ,  G06T19/006

Abstract: Disclosed embodiments pertain to apparatus, systems, and methods for mixed reality. In some embodiments, a camera pose relative to a tracked object in a live image may be determined and used to render synthetic images from keyframes in a 3D model without the tracked object. Optical flow magnitudes for pixels in a first mask region relative to a subset of the synthetic images may be determined and the optical flow magnitudes may be used to determine pixels in each of the subset of synthetic images that correspond to pixels in the first mask. For each pixel in the first mask, a corresponding replacement pixel may be determined as a function of pixels in the subset of synthetic images that correspond to the corresponding pixel in the first mask.

Abstract translation: 公开的实施例涉及用于混合现实的装置，系统和方法。 在一些实施例中，可以确定相对于实时图像中的被跟踪对象的照相机姿态并用于在没有被跟踪对象的3D模型中从关键帧渲染合成图像。 可以确定第一掩模区域中相对于合成图像的子集的像素的光学流量大小，并且可以使用光学流量幅度来确定与第一掩模中的像素对应的合成图像的每个子集中的像素。 对于第一掩模中的每个像素，可以将对应的替换像素确定为与第一掩码中的对应像素对应的合成图像的子集中的像素的函数。

公开(公告)号：US20150262428A1

公开(公告)日：2015-09-17

申请号：US14640981

申请日：2015-03-06

Applicant: QUALCOMM Incorporated

Inventor： Markus Tatzgern ,  Denis Kalkofen ,  Dieter Schmalstieg ,  Raphael David Andre Grasset

IPC: G06T19/00 ,  G06T7/00 ,  G06T17/00

CPC classification number: G06T19/006 ,  G06F3/011 ,  G06F3/017 ,  G06F3/147 ,  G06K9/00671 ,  G06T11/00 ,  G06T2210/61

Abstract: Methods, systems, computer-readable media, and apparatuses for hierarchical clustering for view management in augmented reality are presented. For example one disclosed method includes the steps of accessing point of interest (POI) metadata for a plurality of points of interest associated with a scene; generating a hierarchical cluster tree for at least a portion of the POIs; establishing a plurality of subdivisions associated with the scene; selecting a plurality of POIs from the hierarchical cluster tree for display based on an augmented reality (AR) viewpoint of the scene, the plurality of subdivisions, and a traversal of at least a portion of the hierarchical cluster tree; and displaying labels comprising POI metadata associated with the selected plurality of POIs, the displaying based on placements determined using image-based saliency.

Abstract translation: 提出了用于增强现实中的视图管理的层次聚类的方法，系统，计算机可读介质和装置。 例如，一种公开的方法包括以下步骤：为与场景相关联的多个感兴趣点访问兴趣点（POI）元数据; 为所述POI的至少一部分生成分层聚簇树; 建立与场景相关联的多个细分; 基于所述场景的增强现实（AR）视点，所述多个子部分以及所述分层聚簇树的至少一部分的遍历来从所述分层聚簇树中选择多个POI以进行显示; 以及显示包括与所选择的多个POI相关联的POI元数据的标签，基于使用基于图像的显着性确定的布局进行显示。

公开(公告)号：US20140323148A1

公开(公告)日：2014-10-30

申请号：US14139856

申请日：2013-12-23

Applicant: QUALCOMM Incorporated

Inventor： Dieter Schmalstieg ,  Clemens Arth ,  Johnathan Ventura ,  Christian Pirchheim ,  Gerhard Reitmayr

IPC: H04W4/04 ,  G01V3/38

CPC classification number: H04W4/029 ,  G01C21/005 ,  G01V3/38

Abstract: Exemplary methods, apparatuses, and systems for performing wide area localization from simultaneous localization and mapping (SLAM) maps are disclosed. A mobile device can select a first keyframe based SLAM map of the local environment with one or more received images. A respective localization of the mobile device within the local environment can be determined, and the respective localization may be based on the keyframe based SLAM map. The mobile device can send the first keyframe to a server and receive a first global localization response representing a correction to a local map on the mobile device. The first global localization response can include rotation, translation, and scale information. A server can receive keyframes from a mobile device, and localize the keyframes within a server map by matching keyframe features received from the mobile device to server map features.

Abstract translation: 公开了用于从同时定位和映射（SLAM）映射执行广域定位的示例性方法，装置和系统。 移动设备可以使用一个或多个接收到的图像来选择本地环境的基于关键帧的第一基于SLAM映射。 可以确定移动设备在本地环境中的相应定位，并且相应的定位可以基于基于关键帧的SLAM映射。 移动设备可以将第一关键帧发送到服务器并且接收表示对移动设备上的本地地图的校正的第一全局定位响应。 第一个全球本地化响应可以包括旋转，翻译和缩放信息。 服务器可以从移动设备接收关键帧，并通过将从移动设备接收到的关键帧功能与服务器映射功能进行匹配来本地化服务器映射中的关键帧。

公开(公告)号：US11380047B2

公开(公告)日：2022-07-05

申请号：US16894643

申请日：2020-06-05

Applicant: QUALCOMM Incorporated

Inventor： Philip Voglreiter ,  Dieter Schmalstieg ,  Markus Steinberger

IPC: G06T15/40 ,  G06T15/00 ,  G06T15/20

Abstract: The present disclosure relates to methods and apparatus for graphics processing. The apparatus can determine geometry information for each of a plurality of primitives associated with a viewpoint in a scene. The apparatus can also calculate at least one of surface information and disocclusion information based on the geometry information for each of the plurality of primitives, where the surface information and the disocclusion information may be associated with a volumetric grid based on a viewing area corresponding to the viewpoint. Also, the apparatus can calculate visibility information for each of the plurality of primitives based on at least one of the surface information and the disocclusion information, where the visibility information may be associated with the volumetric grid. The apparatus can also determine whether each of the plurality of primitives is visible based on the visibility information for each of the plurality of primitives.

公开(公告)号：US20190066365A1

公开(公告)日：2019-02-28

申请号：US15683613

申请日：2017-08-22

Applicant: QUALCOMM Incorporated

Inventor： Dieter Schmalstieg ,  Gerhard Reitmayr ,  Serafin Diaz Spindola

IPC: G06T15/40 ,  G06T15/00 ,  G06T15/80 ,  G06T1/60 ,  G06T17/20 ,  G06T15/04

CPC classification number: G06T15/405 ,  A63F13/50 ,  A63F2300/203 ,  A63F2300/8082 ,  G06T1/60 ,  G06T15/005 ,  G06T15/04 ,  G06T15/10 ,  G06T15/80 ,  G06T17/20 ,  G06T2200/16 ,  G06T2200/24

Abstract: An example system includes a first computing device comprising a first graphics processing unit (GPU) implemented in circuitry, and a second computing device comprising a second GPU implemented in circuitry. The first GPU is configured to perform a first portion of an image rendering process to generate intermediate graphics data and send the intermediate graphics data to the second computing device. The second GPU is configured to perform a second portion of the image rendering process to render an image from the intermediate graphics data. The first computing device may be a video game console, and the second computing device may be a virtual reality (VR) headset that warps the rendered image to produce a stereoscopic image pair.

公开(公告)号：US10062210B2

公开(公告)日：2018-08-28

申请号：US14189838

申请日：2014-02-25

Applicant: QUALCOMM Incorporated

Inventor： Lukas Gruber, Jr. ,  Dieter Schmalstieg

IPC: G09G5/00 ,  G06T15/80 ,  G06T15/50 ,  G06T15/60 ,  G06T19/00 ,  G06T15/55 ,  G06T15/06

CPC classification number: G06T19/006 ,  G06T15/06 ,  G06T15/50 ,  G06T15/506 ,  G06T15/55 ,  G06T15/60 ,  G06T15/80 ,  G06T2215/16

Abstract: Methods, systems, computer-readable media, and apparatuses for radiance transfer sampling for augmented reality are presented. In some embodiments, a method includes receiving at least one video frame of an environment. The method further includes generating a surface reconstruction of the environment. The method additionally includes projecting a plurality of rays within the surface reconstruction of the environment. Upon projecting a plurality of rays within the surface reconstruction of the environment, the method includes generating illumination data of the environment from the at least one video frame. The method also includes determining a subset of rays from the plurality of rays in the environment based on areas within the environment needing refinement. The method further includes rendering the virtual object over the video frames based on the plurality of rays excluding the subset of rays.

公开(公告)号：US09934611B2

公开(公告)日：2018-04-03

申请号：US14250544

申请日：2014-04-11

Applicant: QUALCOMM Incorporated

Inventor： Dieter Schmalstieg ,  Gerhard Reitmayr ,  Thanh Quoc Nguyen ,  Raphael David Andre Grasset ,  Tobias Martin Langlotz ,  Hartmut Seichter

IPC: G06T19/00 ,  G06T17/00 ,  H04N13/00 ,  G06T7/12

CPC classification number: G06T19/006 ,  G06T7/12 ,  G06T17/00 ,  G06T2207/10028 ,  G06T2207/20096 ,  G06T2207/20101 ,  H04N13/00

Abstract: Techniques are presented for constructing a digital representation of a physical environment. In some embodiments, a method includes obtaining image data indicative of the physical environment; receiving gesture input data from a user corresponding to at least one location in the physical environment, based on the obtained image data; detecting at least one discontinuity in the physical environment near the at least one location corresponding to the received gesture input data; and generating a digital surface corresponding to a surface in the physical environment, based on the received gesture input data and the at least one discontinuity.

公开(公告)号：US09147221B2

公开(公告)日：2015-09-29

申请号：US13720735

申请日：2012-12-19

Applicant: QUALCOMM INCORPORATED

Inventor： Raphael Grasset ,  Markus Tatzgern ,  Tobias Langlotz ,  Denis Kalkofen ,  Dieter Schmalstieg

IPC: G06T7/00 ,  G06T19/00 ,  G06T1/00 ,  G06T11/00 ,  G06T11/60

CPC classification number: G06T1/00 ,  G06T7/73 ,  G06T11/00 ,  G06T11/60 ,  G06T19/006

Abstract: A mobile device uses an image-driven view management approach for annotating images in real-time. An image-based layout process used by the mobile device computes a saliency map and generates an edge map from a frame of a video stream. The saliency map may be further processed by applying thresholds to reduce the number of saliency levels. The saliency map and edge map are used together to determine a layout position of labels to be rendered over the video stream. The labels are displayed in the layout position until a change of orientation of the camera that exceeds a threshold is detected. Additionally, the representation of the label may be adjusted, e.g., based on a plurality of pixels bounded by an area that is coincident with a layout position for a label in the video frame.

Abstract translation: 移动设备使用图像驱动视图管理方法来实时注释图像。 由移动设备使用的基于图像的布局处理计算显着图并且从视频流的帧生成边缘图。 可以通过应用阈值来进一步处理显着图，以减少显着性水平的数量。 显着性图和边缘图一起用于确定要在视频流上渲染的标签的布局位置。 标签以布局位置显示，直到检测到相机超过阈值的方向改变为止。 此外，可以例如基于由与视频帧中的标签的布局位置重合的区域限定的多个像素来调整标签的表示。

公开(公告)号：US20150262412A1

公开(公告)日：2015-09-17

申请号：US14593949

申请日：2015-01-09

Applicant: QUALCOMM Incorporated

Inventor： Lukas Gruber ,  Dieter Schmalstieg ,  Jonathan Daniel Ventura

IPC: G06T15/50 ,  G06T19/00

CPC classification number: G06T15/50 ,  G06T19/006

Abstract: Methods for determination of AR lighting with dynamic geometry are disclosed. A camera pose for a first image comprising a plurality of pixels may be determined, where each pixel in the first image comprises a depth value and a color value. The first image may correspond to a portion of a 3D model. A second image may be obtained by projecting the portion of the 3D model into a camera field of view based on the camera pose. A composite image comprising a plurality of composite pixels may be obtained based, in part, on the first image and the second image, where each composite pixel in a subset of the plurality of composite pixels is obtained, based, in part, on a corresponding absolute difference between a depth value of a corresponding pixel in the first image and a depth value of a corresponding pixel in the second image.

Abstract translation: 公开了用动态几何测定AR照明的方法。 可以确定包括多个像素的第一图像的相机姿态，其中第一图像中的每个像素包括深度值和颜色值。 第一图像可以对应于3D模型的一部分。 可以通过基于相机姿态将3D模型的部分投影到相机视场中来获得第二图像。 可以部分地基于第一图像和第二图像获得包括多个复合像素的合成图像，其中获得多个复合像素的子集中的每个复合像素，部分地基于对应的 第一图像中的对应像素的深度值与第二图像中的相应像素的深度值之间的绝对差。

公开(公告)号：US20140321702A1

公开(公告)日：2014-10-30

申请号：US14225157

申请日：2014-03-25

Applicant: QUALCOMM Incorporated

Inventor： Dieter Schmalstieg

IPC: G06K9/00

CPC classification number: G06K9/00624 ,  G06T7/20 ,  G06T19/006

Abstract: Disclosed embodiments pertain to apparatus, systems, and methods for mixed reality. In some embodiments, a camera pose relative to a tracked object in a live image may be determined and used to render synthetic images from keyframes in a 3D model without the tracked object. Optical flow magnitudes for pixels in a first mask region relative to a subset of the synthetic images may be determined and the optical flow magnitudes may be used to determine pixels in each of the subset of synthetic images that correspond to pixels in the first mask. For each pixel in the first mask, a corresponding replacement pixel may be determined as a function of pixels in the subset of synthetic images that correspond to the corresponding pixel in the first mask.

Abstract translation: 公开的实施例涉及用于混合现实的装置，系统和方法。 在一些实施例中，可以确定相对于实时图像中的被跟踪对象的照相机姿态并用于在没有被跟踪对象的3D模型中从关键帧渲染合成图像。 可以确定第一掩模区域中相对于合成图像的子集的像素的光学流量大小，并且可以使用光学流量幅度来确定与第一掩模中的像素对应的合成图像的每个子集中的像素。 对于第一掩模中的每个像素，可以将对应的替换像素确定为与第一掩码中的对应像素对应的合成图像的子集中的像素的函数。