公开(公告)号：US20100128112A1

公开(公告)日：2010-05-27

申请号：US12323789

申请日：2008-11-26

申请人： Stefan Marti ,  Francisco Imai ,  Seung Wook Kim

发明人： Stefan Marti ,  Francisco Imai ,  Seung Wook Kim

IPC分类号： H04N13/00 ,  G06K9/00

CPC分类号： G06F3/012 ,  G06F3/011 ,  G06F3/016 ,  H04N13/366

摘要： A system for displaying three-dimensional (3-D) content and enabling a user to interact with the content in an immersive, realistic environment is described. The system has a display component that is non-planar and provides the user with an extended field-of-view (FOV), one factor in the creating the immersive user environment. The system also has a tracking sensor component for tracking a user face. The tracking sensor may include one or more 3-D and 2-D cameras. In addition to tracking the face or head, it may also track other body parts, such as hands and arms. An image perspective adjustment module processes data from the face tracking and enables the user to perceive the 3-D content with motion parallax. The hand and other body part output data is used by gesture detection modules to detect collisions between the user's hand and 3-D content. When a collision is detected, there may be tactile feedback to the user to indicate that there has been contact with a 3-D object. All these components contribute towards creating an immersive and realistic environment for viewing and interacting with 3-D content.

摘要翻译： 描述了一种用于显示三维（3-D）内容并使用户能够在沉浸式，逼真的环境中与内容交互的系统。 该系统具有非平面的显示组件，并为用户提供扩展视野（FOV），这是创建沉浸式用户环境的一个因素。 该系统还具有跟踪用户面部的跟踪传感器部件。 跟踪传感器可以包括一个或多个3-D和2-D相机。 除了跟踪脸部或头部之外，它还可以跟踪其他身体部位，如手和手臂。 图像透视调整模块处理来自面部跟踪的数据，并且使得用户能够利用运动视差来感知3-D内容。 手部和其他身体部位输出数据由手势检测模块用于检测用户手和3-D内容之间的碰撞。 当检测到碰撞时，可能有触觉反馈给用户以指示已经接触到3-D物体。 所有这些组件都有助于创建一个沉浸式和逼真的环境，用于查看和与3D内容进行交互。

公开(公告)号：US08456524B2

公开(公告)日：2013-06-04

申请号：US12699568

申请日：2010-02-03

申请人： Stefan Marti ,  Seung Wook Kim ,  Francisco Imai

发明人： Stefan Marti ,  Seung Wook Kim ,  Francisco Imai

IPC分类号： H04N7/00

CPC分类号： G06T19/006 ,  G06F3/011 ,  G06F3/016 ,  G06F3/017 ,  G06F3/04815 ,  G06F3/1446 ,  G06T2219/2016 ,  G06T2219/2021 ,  G09G2356/00 ,  H04M2203/359 ,  H04N13/279

摘要： Linear and rotational speeds of a mobile device are calculated using distance estimates between imaging sensors in the device and objects or scenes in front of the sensors. The distance estimates are used to modify optical flow vectors from the sensors. Shifting and rotational speeds of the mobile device may then be calculated using the modified optical flow vector values. For example, given a configuration where the first imaging sensor and the second imaging sensor face opposite directions on a single axis, a shifting speed is calculated in the following way: multiplying a first optical flow vector and a first distance estimate, thereby deriving a first modified optical flow vector value; multiplying a second optical flow vector and a second distance estimate, thereby deriving a second modified optical flow vector value; the second modified optical flow vector value may then be subtracted from the first modified optical flow vector value, resulting in a measurement of the shifting speed.

摘要翻译： 使用设备中的成像传感器和传感器前面的对象或场景之间的距离估计来计算移动设备的线性和旋转速度。 距离估计用于修改传感器的光流向量。 然后可以使用修改的光学流向量值来计算移动设备的移动和转速。 例如，给出第一成像传感器和第二成像传感器在单个轴上面对相反方向的配置，以下列方式计算移动速度：将第一光流矢量与第一距离估计相乘，从而得到第一 修改光流向量值; 乘以第二光流矢量和第二距离估计，从而导出第二修改光流矢量值; 然后可以从第一修改的光学流向量值中减去第二修改的光学流向量值，从而得到移位速度的测量值。

公开(公告)号：US20100208029A1

公开(公告)日：2010-08-19

申请号：US12370738

申请日：2009-02-13

申请人： Stefan Marti ,  Francisco Imai ,  Seung Wook Kim

发明人： Stefan Marti ,  Francisco Imai ,  Seung Wook Kim

IPC分类号： H04N7/14 ,  G06F3/041

CPC分类号： H04N7/142 ,  G02B5/30 ,  G02B27/01 ,  G02B27/0101 ,  G02B2027/0138 ,  G02B2027/0156 ,  G02B2027/0187 ,  G06F1/1601

摘要： A mobile content delivery and display system enables a user to use a communication device, such as a cell phone or smart handset device, to view data, images, and video, make phone calls, and perform other functions, in an immersive environment while being mobile. The system, also referred to as a platform, includes a display component which may have one of numerous configurations, each providing extended field-of-views (FOVs). Display component shapes may include hemispherical, ellipsoidal, tubular, conical, pyramidal, or square/rectangular. The display component may have one or more vertical and/or horizontal cuts, each having various degrees of inclination, thereby providing the user with partial physical enclosure creating extended horizontal and/or vertical FOVs. The platform may also have one or more projectors for displaying data (e.g., text, images, or video) on the display component. Other sensors in the system may include 2-D and 3-D cameras, location sensors, speakers, microphones, communication devices, and interfaces. The platform may be worn or attached to the user as an accessory facilitating user mobility.

摘要翻译： 移动内容传递和显示系统使得用户能够使用诸如手机或智能手机设备之类的通信设备在沉浸式环境中查看数据，图像和视频，进行电话呼叫以及执行其他功能，同时 移动。 该系统（也称为平台）包括可以具有多种配置之一的显示组件，每个配置提供扩展视野（FOV）。 显示部件形状可以包括半球形，椭圆形，管状，圆锥形，金字塔形或方形/矩形。 显示组件可以具有一个或多个垂直和/或水平切割，每个具有不同的倾斜度，从而为用户提供部分物理外壳，从而产生扩展的水平和/或垂直FOV。 平台还可以具有用于在显示组件上显示数据（例如，文本，图像或视频）的一个或多个投影仪。 系统中的其他传感器可以包括2-D和3-D摄像机，位置传感器，扬声器，麦克风，通信设备和接口。 该平台可以作为辅助用户移动性的附件佩戴或附接到用户。

公开(公告)号：US20100134618A1

公开(公告)日：2010-06-03

申请号：US12699568

申请日：2010-02-03

申请人： Seung Wook Kim ,  Stefan Marti ,  Francisco Imai

发明人： Seung Wook Kim ,  Stefan Marti ,  Francisco Imai

IPC分类号： H04N7/18

CPC分类号： G06T19/006 ,  G06F3/011 ,  G06F3/016 ,  G06F3/017 ,  G06F3/04815 ,  G06F3/1446 ,  G06T2219/2016 ,  G06T2219/2021 ,  G09G2356/00 ,  H04M2203/359 ,  H04N13/279

摘要： Linear and rotational speeds of a mobile device are calculated using distance estimates between imaging sensors in the device and objects or scenes in front of the sensors. The distance estimates are used to modify optical flow vectors from the sensors. Shifting and rotational speeds of the mobile device may then be calculated using the modified optical flow vector values. For example, given a configuration where the first imaging sensor and the second imaging sensor face opposite directions on a single axis, a shifting speed is calculated in the following way: multiplying a first optical flow vector and a first distance estimate, thereby deriving a first modified optical flow vector value; multiplying a second optical flow vector and a second distance estimate, thereby deriving a second modified optical flow vector value; the second modified optical flow vector value may then be subtracted from the first modified optical flow vector value, resulting in a measurement of the shifting speed.

摘要翻译： 使用设备中的成像传感器和传感器前面的对象或场景之间的距离估计来计算移动设备的线性和旋转速度。 距离估计用于修改传感器的光流向量。 然后可以使用修改的光学流向量值来计算移动设备的移动和转速。 例如，给出第一成像传感器和第二成像传感器在单个轴上面对相反方向的配置，以下列方式计算移动速度：将第一光流矢量与第一距离估计相乘，从而得到第一 修改光流向量值; 乘以第二光流矢量和第二距离估计，从而导出第二修改光流矢量值; 然后可以从第一修改的光学流向量值中减去第二修改的光学流向量值，从而得到移位速度的测量值。

公开(公告)号：US08253795B2

公开(公告)日：2012-08-28

申请号：US12546084

申请日：2009-08-24

申请人： Seung Wook Kim ,  Stefan Marti ,  Francisco Imai

发明人： Seung Wook Kim ,  Stefan Marti ,  Francisco Imai

IPC分类号： H04N7/00

CPC分类号： G06T19/006 ,  G06F3/011 ,  G06F3/016 ,  G06F3/017 ,  G06F3/04815 ,  G06F3/1446 ,  G06T2219/2016 ,  G06T2219/2021 ,  G09G2356/00 ,  H04M2203/359 ,  H04N13/279

摘要： Linear and rotational speeds of a mobile device are calculated using distance estimates between imaging sensors in the device and objects or scenes in front of the sensors. The distance estimates are used to modify optical flow vectors from the sensors. Shifting and rotational speeds of the mobile device may then be calculated using the modified optical flow vector values. For example, given a configuration where the first imaging sensor and the second imaging sensor face opposite directions on a single axis, a shifting speed is calculated in the following way: multiplying a first optical flow vector and a first distance estimate, thereby deriving a first modified optical flow vector value; multiplying a second optical flow vector and a second distance estimate, thereby deriving a second modified optical flow vector value; the second modified optical flow vector value may then be subtracted from the first modified optical flow vector value, resulting in a measurement of the shifting speed.

摘要翻译： 使用设备中的成像传感器和传感器前面的对象或场景之间的距离估计来计算移动设备的线性和旋转速度。 距离估计用于修改传感器的光流向量。 然后可以使用修改的光学流向量值来计算移动设备的移动和转速。 例如，给出第一成像传感器和第二成像传感器在单个轴上面对相反方向的配置，以下列方式计算移动速度：将第一光流矢量与第一距离估计相乘，从而得到第一 修改光流向量值; 乘以第二光流矢量和第二距离估计，从而导出第二修改光流矢量值; 然后可以从第一修改的光学流向量值中减去第二修改的光学流向量值，从而得到移位速度的测量值。

公开(公告)号：US20100053164A1

公开(公告)日：2010-03-04

申请号：US12424117

申请日：2009-04-15

申请人： Francisco Imai ,  Seung Wook Kim ,  Stefan Marti

发明人： Francisco Imai ,  Seung Wook Kim ,  Stefan Marti

IPC分类号： G09G5/12 ,  G06T15/20 ,  G06F13/14

CPC分类号： G06T19/006 ,  G06F3/011 ,  G06F3/016 ,  G06F3/017 ,  G06F3/04815 ,  G06F3/1446 ,  G06T2219/2016 ,  G06T2219/2021 ,  G09G2356/00 ,  H04M2203/359 ,  H04N13/279

摘要： Two or more display components are used to provide spatially correlated displays of 3D content. Three-dimensional content is rendered on multiple displays where the 3D content refers to the same virtual 3D coordinates, in which the relative position of the displays to each other determines the 3D virtual camera position for each display. Although not required, one of the displays may be mobile, such as a cell phone, and the other stationary or nomadic, such as a laptop. Each display shows a view based on a virtual camera into 3D content, such as an online virtual world. By continuously sensing and updating the relative physical distances and orientations of each device to one another, the devices show the user a view into the 3D content that is spatially correlated. Each device has a virtual camera that uses a common pool of 3D geometrical data and renders this data to display images.

摘要翻译： 两个或多个显示组件用于提供3D内容的空间相关显示。 三维内容呈现在多个显示器上，其中3D内容涉及相同的虚拟3D坐标，其中显示器彼此的相对位置确定每个显示器的3D虚拟相机位置。 虽然不是必需的，但是其中一个显示器可以是移动的，例如手机，另一个是静态的或游牧的，例如笔记本电脑。 每个显示器将基于虚拟相机的视图显示为3D内容，例如在线虚拟世界。 通过连续地感测和更新每个设备的相对物理距离和方位彼此之间，设备向用户显示空间相关的3D内容的视图。 每个设备都有一个虚拟相机，它使用公共的三维几何数据池，并呈现这些数据来显示图像。

公开(公告)号：US20100053324A1

公开(公告)日：2010-03-04

申请号：US12546084

申请日：2009-08-24

申请人： Seung Wook Kim ,  Stefan Marti ,  Francisco Imai

发明人： Seung Wook Kim ,  Stefan Marti ,  Francisco Imai

IPC分类号： H04N7/18 ,  G06K9/00

CPC分类号： G06T19/006 ,  G06F3/011 ,  G06F3/016 ,  G06F3/017 ,  G06F3/04815 ,  G06F3/1446 ,  G06T2219/2016 ,  G06T2219/2021 ,  G09G2356/00 ,  H04M2203/359 ,  H04N13/279

摘要： Linear and rotational speeds of a mobile device are calculated using distance estimates between imaging sensors in the device and objects or scenes in front of the sensors. The distance estimates are used to modify optical flow vectors from the sensors. Shifting and rotational speeds of the mobile device may then be calculated using the modified optical flow vector values. For example, given a configuration where the first imaging sensor and the second imaging sensor face opposite directions on a single axis, a shifting speed is calculated in the following way: multiplying a first optical flow vector and a first distance estimate, thereby deriving a first modified optical flow vector value; multiplying a second optical flow vector and a second distance estimate, thereby deriving a second modified optical flow vector value; the second modified optical flow vector value may then be subtracted from the first modified optical flow vector value, resulting in a measurement of the shifting speed.

摘要翻译： 使用设备中的成像传感器和传感器前面的对象或场景之间的距离估计来计算移动设备的线性和旋转速度。 距离估计用于修改传感器的光流向量。 然后可以使用修改的光学流向量值来计算移动设备的移动和转速。 例如，给出第一成像传感器和第二成像传感器在单个轴上面对相反方向的配置，以下列方式计算移动速度：将第一光流矢量与第一距离估计值相乘，从而得到第一 修改光流向量值; 乘以第二光流矢量和第二距离估计，从而导出第二修改光流矢量值; 然后可以从第一修改的光学流向量值中减去第二修改光学流向量值，从而得到移位速度的测量值。

公开(公告)号：US08253649B2

公开(公告)日：2012-08-28

申请号：US12424117

申请日：2009-04-15

申请人： Francisco Imai ,  Seung Wook Kim ,  Stefan Marti

发明人： Francisco Imai ,  Seung Wook Kim ,  Stefan Marti

IPC分类号： G09G5/00 ,  G06T15/00 ,  G06T15/20

CPC分类号： G06T19/006 ,  G06F3/011 ,  G06F3/016 ,  G06F3/017 ,  G06F3/04815 ,  G06F3/1446 ,  G06T2219/2016 ,  G06T2219/2021 ,  G09G2356/00 ,  H04M2203/359 ,  H04N13/279

摘要： Two or more display components are used to provide spatially correlated displays of 3D content. Three-dimensional content is rendered on multiple displays where the 3D content refers to the same virtual 3D coordinates, in which the relative position of the displays to each other determines the 3D virtual camera position for each display. Although not required, one of the displays may be mobile, such as a cell phone, and the other stationary or nomadic, such as a laptop. Each display shows a view based on a virtual camera into 3D content, such as an online virtual world. By continuously sensing and updating the relative physical distances and orientations of each device to one another, the devices show the user a view into the 3D content that is spatially correlated. Each device has a virtual camera that uses a common pool of 3D geometrical data and renders this data to display images.

摘要翻译： 两个或多个显示组件用于提供3D内容的空间相关显示。 三维内容呈现在多个显示器上，其中3D内容涉及相同的虚拟3D坐标，其中显示器彼此的相对位置确定每个显示器的3D虚拟相机位置。 虽然不是必需的，但是其中一个显示器可以是移动的，例如手机，另一个是静态的或游牧的，例如笔记本电脑。 每个显示器将基于虚拟相机的视图显示为3D内容，例如在线虚拟世界。 通过连续地感测和更新每个设备的相对物理距离和方位彼此之间，设备向用户显示空间相关的3D内容的视图。 每个设备都有一个虚拟相机，它使用公共的三维几何数据池，并呈现这些数据来显示图像。

公开(公告)号：US20100053151A1

公开(公告)日：2010-03-04

申请号：US12421363

申请日：2009-04-09

申请人： Stefan Marti ,  Seung Wook Kim ,  Francisco Imai

发明人： Stefan Marti ,  Seung Wook Kim ,  Francisco Imai

IPC分类号： G06T15/00 ,  H04N7/18 ,  H04B3/36

CPC分类号： G06T19/006 ,  G06F3/011 ,  G06F3/016 ,  G06F3/017 ,  G06F3/04815 ,  G06F3/1446 ,  G06F3/147 ,  G06T2219/2016 ,  G06T2219/2021 ,  G09G3/002 ,  G09G2356/00 ,  H04M2203/359 ,  H04N13/279

摘要： A user holds the mobile device upright or sits in front of a nomadic or stationary device, views the monitor from a suitable distance, and physically reaches behind the device with her hand to manipulate a 3D object displayed on the monitor. The device functions as a 3D in-line mediator that provides visual coherency to the user when she reaches behind the device to use hand gestures and movements to manipulate a perceived object behind the device and sees that the 3D object on the display is being manipulated. The perceived object that the user manipulates behind the device with bare hands corresponds to the 3D object displayed on the device. The visual coherency arises from the alignment of the user's head or eyes, the device, and the 3D object. The user's hand may be represented as an image of the actual hand or as a virtualized representation of the hand, such as part of an avatar.

摘要翻译： 用户将移动设备直立或坐在游牧或固定设备的前面，从适当的距离观察监视器，用手操作设备后方来操纵监视器上显示的3D对象。 该设备用作3D在线调解器，当用户到达设备后面时，向用户提供视觉一致性，以使用手势和运动来操纵设备后面的感知对象，并看到显示器上的3D对象正在被操纵。 用户用裸手在设备背后操纵的感知对象对应于设备上显示的3D对象。 视觉一致性源于用户的头部或眼睛，设备和3D对象的对齐。 用户的手可以被表示为实际手的图像或者作为虚拟的手的表示，例如化身的一部分。

公开(公告)号：US08760551B2

公开(公告)日：2014-06-24

申请号：US13039016

申请日：2011-03-02

申请人： John S. Haikin ,  Francisco Imai

发明人： John S. Haikin ,  Francisco Imai

IPC分类号： H04N5/222 ,  H04N5/76

CPC分类号： H04N5/23293 ,  G06K9/00308 ,  H04N1/00167 ,  H04N1/00336 ,  H04N1/00381 ,  H04N1/32101 ,  H04N1/32144 ,  H04N5/23219 ,  H04N5/772 ,  H04N9/8205 ,  H04N9/8227 ,  H04N2101/00 ,  H04N2201/3225 ,  H04N2201/3274

摘要： An image capturing apparatus, method, and storage medium identifying image metadata based on user interest. The image capturing apparatus includes a first image capturing unit configured to capture an image of a scene for a user, a second image capturing unit configured to capture an image of the user, an identification unit configured to identify at least one region of interest of the scene based on a combination of eye and facial characteristics of the user of the image capturing apparatus during an image capturing operation, a processing unit configured to analyze at least facial characteristics of the user associated with each region of interest during the image capturing operation, a determining unit configured to determine a facial expression classification associated with each region of interest based on corresponding analyzed facial characteristics for each region during the image capturing operation, a recording unit configured to record facial expression metadata based on information representing the at least one region of interest and the facial expression classification associated with an image captured during the image capturing operation, and a rendering unit configured to render the image using the recorded facial expression metadata.

摘要翻译： 基于用户兴趣识别图像元数据的图像捕获设备，方法和存储介质。 所述图像拍摄装置包括被配置为捕获用户的场景的图像的第一图像拍摄单元，被配置为捕获用户的图像的第二图像拍摄单元，被配置为识别所述用户的至少一个感兴趣区域的识别单元 在图像拍摄操作期间基于图像捕获装置的用户的眼睛和面部特征的组合的场景，处理单元，被配置为在图像拍摄操作期间至少分析与每个感兴趣区域相关联的用户的面部特征， 确定单元，被配置为基于在图像拍摄操作期间针对每个区域的对应分析的面部特征来确定与每个感兴趣区域相关联的面部表情分类;记录单元，其被配置为基于表示所述至少一个感兴趣区域的信息记录面部表情元数据 和面部表情分类关联 d，其中在图像拍摄操作期间捕获图像，以及呈现单元，被配置为使用记录的面部表情元数据呈现图像。