公开(公告)号：US11780101B2

公开(公告)日：2023-10-10

申请号：US17313921

申请日：2021-05-06

Applicant: MUJIN, Inc.

Inventor： Rosen Diankov ,  Huan Liu ,  Xutao Ye ,  Jose Jeronimo Moreira Rodrigues ,  Yoshiki Kanemoto ,  Jinze Yu ,  Russell Islam

IPC: B25J15/06 ,  B25J9/16 ,  G06T7/13 ,  G06T7/50 ,  B25J13/08 ,  G06T7/00 ,  G06T7/66 ,  B25J9/00 ,  B65G47/91 ,  B65G61/00 ,  G06T1/00 ,  G06V10/44 ,  G06V20/10 ,  G06F18/22

CPC classification number: B25J15/0616 ,  B25J9/0093 ,  B25J9/1697 ,  B25J13/085 ,  B65G47/91 ,  B65G61/00 ,  G06F18/22 ,  G06T1/0014 ,  G06T7/001 ,  G06T7/13 ,  G06T7/50 ,  G06T7/66 ,  G06V10/443 ,  G06V20/10 ,  G05B2219/39001 ,  G05B2219/40067 ,  G06T2207/20212

Abstract: A method of operating a robotic system includes: receiving first image data representative of a package surface; identifying a pair of edges based on the first image data; determining a minimum viable region based on the pair of edges; receiving second image data representative of the package after the lift; and creating registration data based on the first and second image data.

公开(公告)号：US11288814B2

公开(公告)日：2022-03-29

申请号：US16909709

申请日：2020-06-23

Applicant: Mujin, Inc.

Inventor： Jinze Yu ,  Jose Jeronimo Moreira Rodrigues ,  Xutao Ye

IPC: G06T7/181 ,  G06T7/33 ,  G06T7/13 ,  G06T7/174

Abstract: A system and method of detecting objects are provided. The method includes generating first edge information from first image data representing an object based on a first mode of image capture, generating second edge information from second image data representing the object based on a second mode of image capture, the second mode being different from the first mode, fusing the first edge information with the second edge information to generate fused edge information, generating an object detection hypothesis based on the fused edge information, and validating the object detection hypothesis based on the fused edge information, the first edge information, and/or the second edge information.

公开(公告)号：US20220076425A1

公开(公告)日：2022-03-10

申请号：US17527012

申请日：2021-11-15

Applicant: MUJIN, Inc.

Inventor： Jinze Yu ,  Jose Jeronimo Moreira Rodrigues ,  Rosen Nikolaev Diankov ,  Xutao Ye ,  Russell Islam

IPC: G06T7/13 ,  B25J9/16 ,  B65B43/46 ,  B65G61/00 ,  G05B19/4093 ,  G06K9/00 ,  G06K9/32 ,  G06Q10/08 ,  G06T7/11 ,  G06T7/50 ,  G06T7/62 ,  G06T7/73 ,  G06T19/20

Abstract: The present disclosure relates to detecting and registering unrecognized or unregistered objects. A minimum viable range (MVR) may be derived based on inspecting image data that represents objects. The MVR may be determined to be a certain MVR or an uncertain MVR according to one or more features represented in the image data. The MVR may be used to register corresponding objects according to the certain or uncertain determination.

公开(公告)号：US11189033B2

公开(公告)日：2021-11-30

申请号：US16667840

申请日：2019-10-29

Applicant: MUJIN, Inc.

Inventor： Jinze Yu ,  Jose Jeronimo Moreira Rodrigues ,  Rosen Nikolaev Diankov ,  Xutao Ye ,  Russell Islam

IPC: G06F7/00 ,  G06T7/13 ,  B25J9/16 ,  G06T7/73 ,  G06T7/50 ,  G06T7/62 ,  B65G61/00 ,  G05B19/4093 ,  G06Q10/08 ,  G06T19/20 ,  G06T7/11 ,  B65B43/46 ,  G06K9/00 ,  G06K9/32

Abstract: The present disclosure relates to detecting and registering unrecognized or unregistered objects. A minimum viable range (MVR) may be derived based on inspecting image data that represents objects at a start location. The MVR may be determined to be a certain MVR or an uncertain MVR according to one or more features represented in the image data. The MVR may be used to register corresponding objects according to the certain or uncertain determination.

公开(公告)号：US11014241B2

公开(公告)日：2021-05-25

申请号：US16732832

申请日：2020-01-02

Applicant: MUJIN, INC.

Inventor： Russell Islam ,  Xutao Ye ,  Rosen Diankov

IPC: B25J9/16 ,  G06T7/80

Abstract: A robot control system and a method for camera calibration verification is presented. The robot control system is configured to perform a first camera calibration, and to control a robot arm to move a verification symbol to a reference location. The robot control system further receives, from a camera, a reference image of the verification symbol, and determines a reference image coordinate for the verification symbol. The robot control system further controls the robot arm to move the verification symbol to the reference location again during an idle period, receives an additional image of the verification symbol, and determines a verification image coordinate. The robot control system determines a deviation parameter value based the reference image coordinate and the verification image coordinate, and whether the deviation parameter value exceeds a defined threshold, and performs a second camera calibration if the threshold is exceeded.

公开(公告)号：US10987807B2

公开(公告)日：2021-04-27

申请号：US16749354

申请日：2020-01-22

Applicant: Mujin, Inc.

Inventor： Jose Jeronimo Moreira Rodrigues ,  Xutao Ye ,  Jinze Yu ,  Rosen Nikolaev Diankov

IPC: B25J9/16 ,  B65G1/137 ,  G06K9/00 ,  G06K9/62

Abstract: A robotic system includes: a control unit configured to: receive an object set including one or more object entries, wherein: the object entries correspond to source objects of an object source, each of the object entries are described by one or more object entry properties; receive sensor information representing one or more detectable object properties for detectable source objects of the object source; calculate an object match probability between the detectable source objects and the object entries based on a property correlation between the detectable object properties of the detectable source objects and the object entry properties of the object entries; generate an object identity approximation for each of the detectable source objects based on a comparison between the object match probability for each of the detectable source objects corresponding to a particular instance of the object entries; select a target object from the detectable source objects; generate an object handling strategy, for implementation by an object handling unit, to transfer the target object from the object source based on the object entry properties of the object entries corresponding to the object identity approximation; update the object set to indicate that the target object corresponding to a specific instance of the object entries has been removed from the object source; and a storage unit 204, coupled to the control unit, configured to store the object set.

公开(公告)号：US10565737B1

公开(公告)日：2020-02-18

申请号：US16506695

申请日：2019-07-09

Applicant: MUJIN, INC.

Inventor： Russell Islam ,  Xutao Ye

IPC: G06T7/80 ,  H04N13/246 ,  G06T7/73

Abstract: A system and method for performing automatic camera calibration is presented. The system communicates with a first camera and a second camera, wherein a transparent platform is disposed between the two cameras. When a 3D calibration pattern is disposed on the platform, the system receives a first set of calibration images from the first camera, and a second set of calibration images from the second camera. The system determines, based on the first set of calibration images, a first set of coordinates for corners of the polyhedron. The system further determines, based on the second set of calibration images, a second set of coordinates for the corners. The system determines, based on the coordinates, a spatial relationship between the first camera and the second camera. The system further uses a description of the spatial relationship to generate a 3D model of an object other than the 3D calibration pattern.

公开(公告)号：US10399227B1

公开(公告)日：2019-09-03

申请号：US16369630

申请日：2019-03-29

Applicant: MUJIN, INC.

Inventor： Russell Islam ,  Xutao Ye ,  Rosen Diankov

IPC: B25J9/16 ,  G06T7/80

Abstract: A robot control system and a method for camera calibration verification is presented. The robot control system is configured to perform a first camera calibration, and to control a robot arm to move a verification symbol to a reference location. The robot control system further receives, from a camera, a reference image of the verification symbol, and determines a reference image coordinate for the verification symbol. The robot control system further controls the robot arm to move the verification symbol to the reference location again during an idle period, receives an additional image of the verification symbol, and determines a verification image coordinate. The robot control system determines a deviation parameter value based the reference image coordinate and the verification image coordinate, and whether the deviation parameter value exceeds a defined threshold, and performs a second camera calibration if the threshold is exceeded.

公开(公告)号：US10369701B1

公开(公告)日：2019-08-06

申请号：US16290741

申请日：2019-03-01

Applicant: Mujin, Inc.

Inventor： Rosen Diankov ,  Huan Liu ,  Xutao Ye ,  Jose Jeronimo Moreira Rodrigues ,  Yoshiki Kanemoto ,  Jinze Yu ,  Russell Islam

IPC: B25J9/16 ,  G06T7/13 ,  G06T7/50 ,  G06K9/62 ,  B25J13/08 ,  G06T7/00

Abstract: A method of operating a package handling system includes: receiving first image data representative of a first surface image of a package surface; identifying a pair of intersecting edges for the package surface based on the first image; determining a minimum viable region based on the pair of edges, the minimum viable region for gripping and lifting the package; receiving second image data representative of the package after the lift; and creating registration data based on the third image data.

公开(公告)号：US10369698B1

公开(公告)日：2019-08-06

申请号：US16295940

申请日：2019-03-07

Applicant: MUJIN, INC.

Inventor： Russell Islam ,  Xutao Ye ,  Rosen Diankov

IPC: B25J9/16 ,  G06T7/80

Abstract: A robot control system and a method for automatic camera calibration is presented. The robot control system includes a control circuit configured to determine all corner locations of an imaginary cube that fits within a camera field of view, and determine a plurality of locations that are distributed on or throughout the imaginary cube. The control circuit is further configured to control a robot arm to move a calibration pattern to the plurality of locations, and to receive a plurality of calibration images corresponding to the plurality of locations, and to determine respective estimates of intrinsic camera parameters based on the plurality of calibration images, and to determine an estimate of a transformation function that describes a relationship between a camera coordinate system and a world coordinate system. The control circuit is further configured to control placement of the robot arm based on the estimate of the transformation function.