公开(公告)号：US11797926B2

公开(公告)日：2023-10-24

申请号：US17512559

申请日：2021-10-27

Applicant: MUJIN, Inc.

Inventor： Rosen Nikolaev Diankov ,  Russell Islam ,  Xutao Ye

IPC: G06T7/13 ,  G06Q10/087 ,  B25J9/16 ,  G06T7/73 ,  G06T7/50 ,  G06T7/62 ,  B65G61/00 ,  G05B19/4093 ,  G06T19/20 ,  G06T7/11 ,  B65B43/46 ,  G06V10/25 ,  G06V20/64 ,  G06V10/20 ,  G06V10/44

CPC classification number: G06Q10/087 ,  B25J9/1676 ,  B25J9/1687 ,  B25J9/1697 ,  B65B43/46 ,  B65G61/00 ,  G05B19/40935 ,  G06T7/11 ,  G06T7/13 ,  G06T7/50 ,  G06T7/62 ,  G06T7/73 ,  G06T19/20 ,  G06V10/25 ,  G06V10/255 ,  G06V20/64 ,  G05B2219/40006 ,  G05B2219/40607 ,  G06T2200/04 ,  G06T2207/10028 ,  G06T2207/20164 ,  G06T2207/30108 ,  G06V10/44

Abstract: A system and method for operating a robotic system to register unrecognized objects is disclosed. The robotic system may use first image data representative of an unrecognized object to derive an initial minimum viable region (MVR). The robotic system may analyze second image data representative of the unrecognized object to detect a condition representative of an accuracy of the initial MVR. The robotic system may register the initial MVR or an adjustment thereof based on the detected condition.

公开(公告)号：US11688089B2

公开(公告)日：2023-06-27

申请号：US17360073

申请日：2021-06-28

Applicant: MUJIN, INC.

Inventor： Russell Islam ,  Xutao Ye

IPC: G06T7/55 ,  G06T7/70 ,  H04N5/247 ,  G06T7/521 ,  B25J9/16

CPC classification number: G06T7/55 ,  B25J9/1697 ,  G06T7/70 ,  H04N5/247 ,  G06T7/521 ,  G06T2207/10028

Abstract: A method and system for processing camera images is presented. The system receives a first depth map generated based on information sensed by a first type of depth-sensing camera, and receives a second depth map generated based on information sensed by a second type of depth-sensing camera. The first depth map includes a first set of pixels that indicate a first set of respective depth values. The second depth map includes a second set of pixels that indicate a second set of respective depth values. The system identifies a third set of pixels of the first depth map that correspond to the second set of pixels of the second depth map, identifies one or more empty pixels from the third set of pixels, and updates the first depth map by assigning to each empty pixel a respective depth value based on the second depth map.

公开(公告)号：US11074722B2

公开(公告)日：2021-07-27

申请号：US16682560

申请日：2019-11-13

Applicant: MUJIN, INC.

Inventor： Russell Islam ,  Xutao Ye

IPC: H04N13/246 ,  G06T7/80 ,  G06T7/73

Abstract: A system and method for performing automatic camera calibration is presented. The system communicates with a first camera and a second camera, wherein a transparent platform is disposed between the two cameras. When a 3D calibration pattern is disposed on the platform, the system receives a first set of calibration images from the first camera, and a second set of calibration images from the second camera. The system determines, based on the first set of calibration images, a first set of coordinates for corners of the polyhedron. The system further determines, based on the second set of calibration images, a second set of coordinates for the corners. The system determines, based on the coordinates, a spatial relationship between the first camera and the second camera. The system further uses a description of the spatial relationship to generate a 3D model of an object other than the 3D calibration pattern.

公开(公告)号：US10703584B2

公开(公告)日：2020-07-07

申请号：US16539790

申请日：2019-08-13

Applicant: MUJIN, Inc.

Inventor： Rosen Nikolaev Diankov ,  Huan Liu ,  Xutao Ye ,  Jose Jeronimo Moreira Rodrigues ,  Yoshiki Kanemoto ,  Jinze Yu ,  Russell Islam

IPC: B65G61/00 ,  B25J9/16 ,  G06Q10/08 ,  G05B19/4093

Abstract: A method for operating a robotic system includes obtaining and processing first data representative of an object at a start location; obtaining additional data as the object is transferred to a task location, wherein the additional data includes information regarding one or more edges and/or one or more surfaces separate from portions of the object captured in the first data; and creating registration data representative of the object based on the additional data.

公开(公告)号：US10562186B1

公开(公告)日：2020-02-18

申请号：US16525004

申请日：2019-07-29

Applicant: MUJIN, INC.

Inventor： Russell Islam ,  Xutao Ye ,  Rosen Diankov

IPC: B25J9/16 ,  G06T7/80

Abstract: A robot control system and a method for camera calibration verification is presented. The robot control system is configured to perform a first camera calibration, and to control a robot arm to move a verification symbol to a reference location. The robot control system further receives, from a camera, a reference image of the verification symbol, and determines a reference image coordinate for the verification symbol. The robot control system further controls the robot arm to move the verification symbol to the reference location again during an idle period, receives an additional image of the verification symbol, and determines a verification image coordinate. The robot control system determines a deviation parameter value based the reference image coordinate and the verification image coordinate, and whether the deviation parameter value exceeds a defined threshold, and performs a second camera calibration if the threshold is exceeded.

公开(公告)号：US11967113B2

公开(公告)日：2024-04-23

申请号：US17350006

申请日：2021-06-17

Applicant: MUJIN, INC.

Inventor： Russell Islam ,  Xutao Ye

IPC: G06T7/80 ,  G06T7/73 ,  H04N13/246

CPC classification number: G06T7/85 ,  G06T7/75 ,  H04N13/246 ,  G06T2207/10012

Abstract: A system and method for performing automatic camera calibration is presented. The system communicates with a first camera and a second camera, wherein a transparent platform is disposed between the two cameras. When a 3D calibration pattern is disposed on the platform, the system receives a first set of calibration images from the first camera, and a second set of calibration images from the second camera. The system determines, based on the first set of calibration images, a first set of coordinates for corners of the polyhedron. The system further determines, based on the second set of calibration images, a second set of coordinates for the corners. The system determines, based on the coordinates, a spatial relationship between the first camera and the second camera. The system further uses a description of the spatial relationship to generate a 3D model of an object other than the 3D calibration pattern.

公开(公告)号：US20230415360A1

公开(公告)日：2023-12-28

申请号：US18463233

申请日：2023-09-07

Applicant: MUJIN, Inc.

Inventor： Rosen Diankov ,  Huan Liu ,  Xutao Ye ,  Jose Jeronimo Moreira Rodrigues ,  Yoshiki Kanemoto ,  Jinze Yu ,  Russell Islam

IPC: B25J15/06 ,  B25J9/16 ,  G06T7/13 ,  G06T7/50 ,  B25J13/08 ,  G06T7/00 ,  G06T7/66 ,  B25J9/00 ,  B65G47/91 ,  B65G61/00 ,  G06T1/00 ,  G06V10/44 ,  G06V20/10 ,  G06F18/22

CPC classification number: B25J15/0616 ,  B25J9/1697 ,  G06T7/13 ,  G06T7/50 ,  B25J13/085 ,  G06T7/001 ,  G06T7/66 ,  B25J9/0093 ,  B65G47/91 ,  B65G61/00 ,  G06T1/0014 ,  G06V10/443 ,  G06V20/10 ,  G06F18/22 ,  G05B2219/40067 ,  G06T2207/20212 ,  G05B2219/39001

Abstract: A method of operating a robotic system includes: receiving first image data representative of a package surface; identifying a pair of edges based on the first image data; determining a minimum viable region based on the pair of edges; receiving second image data representative of the package after the lift; and creating registration data based on the first and second image data.

公开(公告)号：US11590656B2

公开(公告)日：2023-02-28

申请号：US17135261

申请日：2020-12-28

Applicant: MUJIN, INC.

Inventor： Russell Islam ,  Xutao Ye ,  Rosen Diankov

IPC: B25J9/16 ,  G06T7/80

Abstract: A computing system and a method for calibration verification is presented. The computing system is configured to perform a first calibration operation, and to control a robot arm to move a verification symbol to a reference location. The robot control system further receives, from a camera, a reference image of the verification symbol, and determines a reference image coordinate for the verification symbol. The robot control system further controls the robot arm to move the verification symbol to the reference location again during an idle period, receives an additional image of the verification symbol, and determines a verification image coordinate. The robot control system determines a deviation parameter value based the reference image coordinate and the verification image coordinate, and whether the deviation parameter value exceeds a defined threshold, and performs a second calibration operation if the threshold is exceeded.

公开(公告)号：US20230008540A1

公开(公告)日：2023-01-12

申请号：US17944117

申请日：2022-09-13

Applicant: MUJIN, Inc.

Inventor： Rosen Nikolaev Diankov ,  Huan Liu ,  Xutao Ye ,  Jose Jeronimo Moreira Rodrigues ,  Yoshiki Kanemoto ,  Jinze Yu ,  Russell Islam

IPC: G06T7/13 ,  B25J9/16 ,  G06Q10/08 ,  G06T7/11 ,  G06T7/50 ,  G06V10/25 ,  G06T19/20 ,  G06T7/62 ,  B65G61/00 ,  G06V20/64 ,  G05B19/4093 ,  B65B43/46 ,  G06T7/73

Abstract: A system and method for operating a robotic system to scan and register unrecognized objects is disclosed. The robotic system may use an image data representative of an unrecognized object located at a start location to implement operations for transferring the unrecognized object from the start location. While implementing the operations, the robotic system may obtain additional data, including scanning results of one or more portions of the unrecognized object not included in the image data. The robotic system may use the additional data to register the unrecognized object.

公开(公告)号：US11370121B2

公开(公告)日：2022-06-28

申请号：US16667170

申请日：2019-10-29

Applicant: MUJIN, INC.

Inventor： Russell Islam ,  Puttichai Lertkultanon

IPC: B25J9/16

Abstract: A method and system for determining poses for camera calibration is presented. The system determines a range of pattern orientations for performing the camera calibration, and determines a surface region on a surface of an imaginary sphere, which represents possible pattern orientations for the calibration pattern. The system determines a plurality of poses for the calibration pattern to adopt. The plurality of poses may be defined by respective combinations of a plurality of respective locations within the camera field of view and a plurality of respective sets of pose angle values. Each set of pose angle values of the plurality of respective sets may be based on a respective surface point selected from within the surface region on the surface of the imaginary sphere. The system outputs a plurality of robot movement commands based on the plurality of poses that are determined.