公开(公告)号：US20230281930A1

公开(公告)日：2023-09-07

申请号：US18144058

申请日：2023-05-05

Applicant: Snap Inc.

Inventor： Kyle Goodrich ,  Samuel Edward Hare ,  Maxim Maximov Lazarov ,  Tony Mathew ,  Andrew James McPhee ,  Daniel Moreno ,  Dhritiman Sagar ,  Wentao Shang

IPC: G06T19/00 ,  G06F3/04842 ,  G06T7/50 ,  H04L51/42

CPC classification number: G06T19/00 ,  G06F3/04842 ,  G06T7/50 ,  H04L51/42 ,  G06T2207/20084

Abstract: The subject technology generates depth data using a machine learning model based at least in part on captured image data from at least one camera of a client device. The subject technology applies, to the captured image data and the generated depth data, a 3D effect based at least in part on an augmented reality content generator. The subject technology generates a depth map using at least the depth data. The subject technology generates a packed depth map based at least in part on the depth map, the generating the packed depth map. The subject technology converts a single channel floating point texture to a raw depth map. The subject technology generates multiple channels based at least in part on the raw depth map. The subject technology generates a segmentation mask based at least on the captured image data. The subject technology performs background inpainting and blurring of the captured image data using at least the segmentation mask to generate background inpainted image data.

公开(公告)号：US11750767B2

公开(公告)日：2023-09-05

申请号：US17644501

申请日：2021-12-15

Applicant: Snap Inc.

Inventor： Ebony James Charlton ,  Michael John Evans ,  Samuel Edward Hare ,  Andrew James McPhee ,  Robert Cornelius Murphy ,  Eitan Pilipski

IPC: H04N5/272 ,  G11B27/34 ,  G11B27/11 ,  H04N23/62 ,  H04N23/63

CPC classification number: H04N5/272 ,  G11B27/11 ,  G11B27/34 ,  H04N23/62 ,  H04N23/631 ,  H04N23/635

Abstract: Systems, devices, media and methods are presented for presentation of modified objects within a video stream. The systems and methods select an object of interest depicted within a user interface based on an associated image modifier, determine a modifier context based at least in part on one or more characteristics of the selected object, identify a set of image modifiers based on the modifier context, rank a first portion of the identified set of image modifiers based on a primary ordering characteristic, rank a second portion of the identified set of image modifiers based on a secondary ordering characteristic and cause presentation of the modifier icons for the ranked set of image modifiers.

公开(公告)号：US11743426B2

公开(公告)日：2023-08-29

申请号：US16992968

申请日：2020-08-13

Applicant: Snap Inc.

Inventor： Lidiia Bogdanovych ,  William Brendel ,  Samuel Edward Hare ,  Fedir Poliakov ,  Guohui Wang ,  Xuehan Xiong ,  Jianchao Yang ,  Linjie Yang

IPC: G06T7/194 ,  G06V10/82 ,  H04N7/14 ,  G06T7/11 ,  G06N3/08 ,  G06N3/04 ,  G06V30/242 ,  G06F18/214 ,  G06F18/24 ,  G06V30/19 ,  H04N5/445 ,  H04N5/76

CPC classification number: H04N7/147 ,  G06F18/214 ,  G06F18/24765 ,  G06N3/04 ,  G06N3/08 ,  G06T7/11 ,  G06T7/194 ,  G06V10/82 ,  G06V30/19173 ,  G06V30/242 ,  G06T2207/10016 ,  G06T2207/10024 ,  G06T2207/20024 ,  G06T2207/20081 ,  G06T2207/20084 ,  G06T2207/20221 ,  G06T2207/30201 ,  H04N5/44504 ,  H04N5/76 ,  H04N7/141

Abstract: A machine learning system can generate an image mask (e.g., a pixel mask) comprising pixel assignments for pixels. The pixels can he assigned to classes, including, for example, face, clothes, body skin, or hair. The machine learning system can be implemented. using a convolutional neural network that is configured to execute efficiently on computing devices having limited resources, such as mobile phones. The pixel mask can be used to more accurately display video effects interacting with a user or subject depicted in the image.

公开(公告)号：US11704878B2

公开(公告)日：2023-07-18

申请号：US17505370

申请日：2021-10-19

Applicant: Snap Inc.

Inventor： Andrew James McPhee ,  Ebony James Charlton ,  Samuel Edward Hare ,  Michael John Evans ,  Jokubas Dargis ,  Ricardo Sanchez-Saez

IPC: G06T19/00 ,  G06T15/20 ,  G06T15/60 ,  G06F3/0346 ,  G06F3/01

CPC classification number: G06T19/006 ,  G06F3/011 ,  G06F3/0346 ,  G06T15/205 ,  G06T15/60

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for rendering three-dimensional virtual objects within real-world environments. Virtual rendering of a three-dimensional virtual object can be altered appropriately as a user moves around the object in the real-world through utilization of a redundant tracking system comprising multiple tracking sub-systems. Virtual object rendering can be with respect to a reference surface in a real-world three-dimensional space depicted in a camera view of a mobile computing device.

公开(公告)号：US20230073310A1

公开(公告)日：2023-03-09

申请号：US18056099

申请日：2022-11-16

Applicant: Snap Inc.

Inventor： Kyle Goodrich ,  Samuel Edward Hare ,  Maxim Maximov Lazarov ,  Tony Mathew ,  Andrew James McPhee ,  Daniel Moreno ,  Wentao Shang

IPC: G06T19/00 ,  G06F3/0482 ,  G06F3/0488 ,  G06T5/00 ,  G06F3/04815

Abstract: The subject technology selects a set of augmented reality content generators from available augmented reality content generators, the selected set of augmented reality content generators comprising at least one augmented reality content generator for applying a three-dimensional (3D) effect. The subject technology causes display of a carousel interface including selectable graphical items, each selectable graphical item corresponding to a respective augmented reality content generator. The subject technology receives a selection of a first selectable graphical item from the selectable graphical items, the first selectable graphical item including a first augmented reality content generator for applying a first 3D effect. The subject technology applies, to first image data and first depth data, the first augmented reality content generator corresponding to the selected first selectable graphical item. The subject technology generates a message including the applied first augmented reality content generator to the first image data and the first depth data.

公开(公告)号：US20230005233A1

公开(公告)日：2023-01-05

申请号：US17878635

申请日：2022-08-01

Applicant: Snap Inc.

Inventor： Kyle Goodrich ,  Samuel Edward Hare ,  Maxim Maximov Lazarov ,  Tony Mathew ,  Andrew James McPhee ,  Daniel Moreno ,  Dhritiman Sagar ,  Wentao Shang

IPC: G06T19/20 ,  G06N20/00 ,  G06F3/04842 ,  G06F3/04883 ,  G06T15/50 ,  G06T7/507 ,  G06V40/16

Abstract: The subject technology receives a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator for applying a 3D effect, the 3D effect including at least one beautification operation. The subject technology captures image data and depth data using a camera. The subject technology applies, to the image data and the depth data, the 3D effect including the at least one beautification operation based at least in part on the augmented reality content generator, the beautification operation being performed as part of applying the 3D effect. The subject technology generates a 3D message based at least in part on the applied 3D effect including the at least one beautification operation. The subject technology renders a view of the 3D message based at least in part on the applied 3D effect including the at least one beautification operation.

公开(公告)号：US11443491B2

公开(公告)日：2022-09-13

申请号：US17249092

申请日：2021-02-19

Applicant: Snap Inc.

Inventor： Samuel Edward Hare ,  Andrew James McPhee ,  Maxim Maximov Lazarov ,  Wentao Shang ,  Kyle Goodrich ,  Tony Mathew

IPC: G06T19/00 ,  G06T19/20 ,  H04L51/04 ,  H04N5/247 ,  H04N5/272

Abstract: Systems and methods are provided for capturing by a camera of a user device, a first image depicting a first environment of the user device; overlaying a first virtual object on a portion of the first image depicting the first environment; modifying a surface of the first virtual object using content captured by the user device; storing a second virtual object comprising the first virtual object with the modified surface; and generating for display the second virtual object on a portion of a second image depicting a second environment.

公开(公告)号：US11410401B2

公开(公告)日：2022-08-09

申请号：US17006471

申请日：2020-08-28

Applicant: Snap Inc.

Inventor： Kyle Goodrich ,  Samuel Edward Hare ,  Maxim Maximov Lazarov ,  Tony Mathew ,  Andrew James McPhee ,  Daniel Moreno ,  Dhritiman Sagar ,  Wentao Shang

IPC: G06T19/20 ,  G06N20/00 ,  G06F3/04842 ,  G06F3/04883 ,  G06T15/50 ,  G06T7/507 ,  G06V40/16

Abstract: The subject technology receives a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator for applying a 3D effect, the 3D effect including at least one beautification operation. The subject technology captures image data and depth data using a camera. The subject technology applies, to the image data and the depth data, the 3D effect including the at least one beautification operation based at least in part on the augmented reality content generator, the beautification operation being performed as part of applying the 3D effect. The subject technology generates a 3D message based at least in part on the applied 3D effect including the at least one beautification operation. The subject technology renders a view of the 3D message based at least in part on the applied 3D effect including the at least one beautification operation.

公开(公告)号：US20220172448A1

公开(公告)日：2022-06-02

申请号：US17651524

申请日：2022-02-17

Applicant: Snap Inc.

Inventor： Travis Chen ,  Samuel Edward Hare ,  Yuncheng Li ,  Tony Mathew ,  Jonathan Solichin ,  Jianchao Yang ,  Ning Zhang

IPC: G06T19/20 ,  G06T7/20 ,  G06T19/00 ,  G06T7/73 ,  G06V10/20 ,  G06V20/20 ,  G06V20/40

Abstract: Systems, devices, media, and methods are presented for object detection and inserting graphical elements into an image stream in response to detecting the object. The systems and methods detect an object of interest in received frames of a video stream. The systems and methods identify a bounding box for the object of interest and estimate a three-dimensional position of the object of interest based on a scale of the object of interest. The systems and methods generate one or more graphical elements having a size based on the scale of the object of interest and a position based on the three-dimensional position estimated for the object of interest. The one or more graphical elements are generated within the video stream to form a modified video stream. The systems and methods cause presentation of the modified video stream including the object of interest and the one or more graphical elements.

公开(公告)号：US11233952B2

公开(公告)日：2022-01-25

申请号：US16815804

申请日：2020-03-11

Applicant: Snap Inc.

Inventor： Ebony James Charlton ,  Michael John Evans ,  Samuel Edward Hare ,  Andrew James McPhee ,  Robert Cornelius Murphy ,  Eitan Pilipski

IPC: H04N5/272 ,  G11B27/34 ,  G11B27/11 ,  H04N5/232

Abstract: Systems, devices, media and methods are presented for presentation of modified objects within a video stream. The systems and methods select an object of interest depicted within a user interface based on an associated image modifier, determine a modifier context based at least in part on one or more characteristics of the selected object, identify a set of image modifiers based on the modifier context, rank a first portion of the identified set of image modifiers based on a primary ordering characteristic, rank a second portion of the identified set of image modifiers based on a secondary ordering characteristic and cause presentation of the modifier icons for the ranked set of image modifiers.