公开(公告)号：US12283064B2

公开(公告)日：2025-04-22

申请号：US18526261

申请日：2023-12-01

Applicant: MUJIN, INC.

Inventor： Rosen Diankov ,  Xutao Ye ,  Ziyan Zhou

IPC: G06T7/55 ,  B25J9/00 ,  B25J9/16 ,  B65G59/02 ,  B65G67/02 ,  B65G67/04 ,  B65G67/24 ,  G06T5/50 ,  G06T7/50 ,  G06T7/593

Abstract: A control apparatus includes a first information acquiring section that acquires three-dimensional information of a first region of surfaces of a plurality of objects, the information being obtained by imaging or scanning the plurality of objects from a first location; a second information acquiring section that acquires three-dimensional information of a second region of surfaces of the plurality of objects, the information being obtained by imaging or scanning the plurality of objects from a second location; and a combining section that generates information indicating three-dimensional shapes of at least a portion of the surfaces of the plurality of objects, based on the three-dimensional information of the first region acquired by the first information acquiring section and the three-dimensional information of the second region acquired by the second information acquiring section.

公开(公告)号：US11883966B2

公开(公告)日：2024-01-30

申请号：US17116436

申请日：2020-12-09

Applicant: Mujin, Inc.

Inventor： Xutao Ye ,  Puttichai Lertkultanon ,  Rosen Nikolaev Diankov

IPC: B25J19/02 ,  B25J9/16 ,  G06T7/73 ,  B25J13/08 ,  B25J15/00 ,  B65G59/02 ,  G05B19/4155 ,  G06T7/60 ,  G06F18/2413 ,  H04N23/54 ,  H04N23/695 ,  G06V10/764 ,  G06V20/10

CPC classification number: B25J9/1697 ,  B25J9/1612 ,  B25J9/1653 ,  B25J9/1664 ,  B25J9/1669 ,  B25J9/1671 ,  B25J13/08 ,  B25J15/0061 ,  B25J19/023 ,  B65G59/02 ,  G05B19/4155 ,  G06F18/2413 ,  G06T7/60 ,  G06T7/74 ,  G06V10/764 ,  G06V20/10 ,  H04N23/54 ,  H04N23/695 ,  G05B2219/40269 ,  G06T2207/10028 ,  G06T2207/20164 ,  G06T2207/30244

Abstract: A method and computing system for performing object detection are presented. The computing system may be configured to: receive first image information that represents at least a first portion of an object structure of an object in a camera's field of view, wherein the first image information is associate with a first camera pose; generate or update, based on the first image information, sensed structure information representing the object structure; identify an object corner associated with the object structure; cause the robot arm to move the camera to a second camera pose in which the camera is pointed at the object corner; receive second image information associated with the second camera pose; update the sensed structure information based on the second image information; determine, based on the updated sensed structure information, an object type associated with the object; determine one or more robot interaction locations based on the object type.

公开(公告)号：US11797926B2

公开(公告)日：2023-10-24

申请号：US17512559

申请日：2021-10-27

Applicant: MUJIN, Inc.

Inventor： Rosen Nikolaev Diankov ,  Russell Islam ,  Xutao Ye

IPC: G06T7/13 ,  G06Q10/087 ,  B25J9/16 ,  G06T7/73 ,  G06T7/50 ,  G06T7/62 ,  B65G61/00 ,  G05B19/4093 ,  G06T19/20 ,  G06T7/11 ,  B65B43/46 ,  G06V10/25 ,  G06V20/64 ,  G06V10/20 ,  G06V10/44

CPC classification number: G06Q10/087 ,  B25J9/1676 ,  B25J9/1687 ,  B25J9/1697 ,  B65B43/46 ,  B65G61/00 ,  G05B19/40935 ,  G06T7/11 ,  G06T7/13 ,  G06T7/50 ,  G06T7/62 ,  G06T7/73 ,  G06T19/20 ,  G06V10/25 ,  G06V10/255 ,  G06V20/64 ,  G05B2219/40006 ,  G05B2219/40607 ,  G06T2200/04 ,  G06T2207/10028 ,  G06T2207/20164 ,  G06T2207/30108 ,  G06V10/44

Abstract: A system and method for operating a robotic system to register unrecognized objects is disclosed. The robotic system may use first image data representative of an unrecognized object to derive an initial minimum viable region (MVR). The robotic system may analyze second image data representative of the unrecognized object to detect a condition representative of an accuracy of the initial MVR. The robotic system may register the initial MVR or an adjustment thereof based on the detected condition.

公开(公告)号：US11717971B2

公开(公告)日：2023-08-08

申请号：US17385349

申请日：2021-07-26

Applicant: MUJIN, INC.

Inventor： Xutao Ye ,  Puttichai Lertkultanon ,  Rosen Nikolaev Diankov

IPC: G06F7/00 ,  B25J9/16 ,  B25J19/02 ,  G06T7/73 ,  B25J13/08 ,  B25J15/00 ,  B65G59/02 ,  G05B19/4155 ,  G06T7/60 ,  G06F18/2413 ,  H04N23/54 ,  H04N23/695 ,  G06V10/764 ,  G06V20/10

CPC classification number: B25J9/1697 ,  B25J9/1612 ,  B25J9/1653 ,  B25J9/1664 ,  B25J9/1669 ,  B25J9/1671 ,  B25J13/08 ,  B25J15/0061 ,  B25J19/023 ,  B65G59/02 ,  G05B19/4155 ,  G06F18/2413 ,  G06T7/60 ,  G06T7/74 ,  G06V10/764 ,  G06V20/10 ,  H04N23/54 ,  H04N23/695 ,  G05B2219/40269 ,  G06T2207/10028 ,  G06T2207/20164 ,  G06T2207/30244

Abstract: A system and method for motion planning is presented. The system is configured, when an object is or has been in a camera field of view of a camera, to receive first image information that is generated when the camera has a first camera pose. The system is further configured to determine, based on the first image information, a first estimate of the object structure, and to identify, based on the first estimate of the object structure or based on the first image information, an object corner. The system is further configured to cause an end effector apparatus to move the camera to a second camera pose, and to receive second image information for representing the object's structure. The system is configured to determine a second estimate of the object's structure based on the second image information, and to generate a motion plan based on at least the second estimate.

公开(公告)号：US11688089B2

公开(公告)日：2023-06-27

申请号：US17360073

申请日：2021-06-28

Applicant: MUJIN, INC.

Inventor： Russell Islam ,  Xutao Ye

IPC: G06T7/55 ,  G06T7/70 ,  H04N5/247 ,  G06T7/521 ,  B25J9/16

CPC classification number: G06T7/55 ,  B25J9/1697 ,  G06T7/70 ,  H04N5/247 ,  G06T7/521 ,  G06T2207/10028

Abstract: A method and system for processing camera images is presented. The system receives a first depth map generated based on information sensed by a first type of depth-sensing camera, and receives a second depth map generated based on information sensed by a second type of depth-sensing camera. The first depth map includes a first set of pixels that indicate a first set of respective depth values. The second depth map includes a second set of pixels that indicate a second set of respective depth values. The system identifies a third set of pixels of the first depth map that correspond to the second set of pixels of the second depth map, identifies one or more empty pixels from the third set of pixels, and updates the first depth map by assigning to each empty pixel a respective depth value based on the second depth map.

公开(公告)号：US11103998B2

公开(公告)日：2021-08-31

申请号：US17084272

申请日：2020-10-29

Applicant: MUJIN, INC.

Inventor： Xutao Ye ,  Puttichai Lertkultanon ,  Rosen Nikolaev Diankov

IPC: G06F7/00 ,  B25J9/16 ,  B25J13/08 ,  B25J15/00 ,  B65G59/02 ,  G05B19/4155 ,  H04N5/232 ,  G06T7/60 ,  G06K9/62 ,  H04N5/225

Abstract: A system and method for motion planning is presented. The system is configured, when an object is or has been in a camera field of view of a camera, to receive first image information that is generated when the camera has a first camera pose. The system is further configured to determine, based on the first image information, a first estimate of the object structure, and to identify, based on the first estimate of the object structure or based on the first image information, an object corner. The system is further configured to cause an end effector apparatus to move the camera to a second camera pose, and to receive second image information for representing the object's structure. The system is configured to determine a second estimate of the object's structure based on the second image information, and to generate a motion plan based on at least the second estimate.

公开(公告)号：US11074722B2

公开(公告)日：2021-07-27

申请号：US16682560

申请日：2019-11-13

Applicant: MUJIN, INC.

Inventor： Russell Islam ,  Xutao Ye

IPC: H04N13/246 ,  G06T7/80 ,  G06T7/73

Abstract: A system and method for performing automatic camera calibration is presented. The system communicates with a first camera and a second camera, wherein a transparent platform is disposed between the two cameras. When a 3D calibration pattern is disposed on the platform, the system receives a first set of calibration images from the first camera, and a second set of calibration images from the second camera. The system determines, based on the first set of calibration images, a first set of coordinates for corners of the polyhedron. The system further determines, based on the second set of calibration images, a second set of coordinates for the corners. The system determines, based on the coordinates, a spatial relationship between the first camera and the second camera. The system further uses a description of the spatial relationship to generate a 3D model of an object other than the 3D calibration pattern.

公开(公告)号：US20200316774A1

公开(公告)日：2020-10-08

申请号：US16749354

申请日：2020-01-22

Applicant: Mujin, Inc.

Inventor： Jose Jeronimo Moreira Rodrigues ,  Xutao Ye ,  Jinze Yu ,  Rosen Nikolaev Diankov

IPC: B25J9/16 ,  B65G1/137

Abstract: A robotic system includes: a control unit configured to: receive an object set including one or more object entries, wherein: the object entries correspond to source objects of an object source, each of the object entries are described by one or more object entry properties; receive sensor information representing one or more detectable object properties for detectable source objects of the object source; calculate an object match probability between the detectable source objects and the object entries based on a property correlation between the detectable object properties of the detectable source objects and the object entry properties of the object entries; generate an object identity approximation for each of the detectable source objects based on a comparison between the object match probability for each of the detectable source objects corresponding to a particular instance of the object entries; select a target object from the detectable source objects; generate an object handling strategy, for implementation by an object handling unit, to transfer the target object from the object source based on the object entry properties of the object entries corresponding to the object identity approximation; update the object set to indicate that the target object corresponding to a specific instance of the object entries has been removed from the object source; and a storage unit 204, coupled to the control unit, configured to store the object set.

公开(公告)号：US10703584B2

公开(公告)日：2020-07-07

申请号：US16539790

申请日：2019-08-13

Applicant: MUJIN, Inc.

Inventor： Rosen Nikolaev Diankov ,  Huan Liu ,  Xutao Ye ,  Jose Jeronimo Moreira Rodrigues ,  Yoshiki Kanemoto ,  Jinze Yu ,  Russell Islam

IPC: B65G61/00 ,  B25J9/16 ,  G06Q10/08 ,  G05B19/4093

Abstract: A method for operating a robotic system includes obtaining and processing first data representative of an object at a start location; obtaining additional data as the object is transferred to a task location, wherein the additional data includes information regarding one or more edges and/or one or more surfaces separate from portions of the object captured in the first data; and creating registration data representative of the object based on the additional data.

公开(公告)号：US10562186B1

公开(公告)日：2020-02-18

申请号：US16525004

申请日：2019-07-29

Applicant: MUJIN, INC.

Inventor： Russell Islam ,  Xutao Ye ,  Rosen Diankov

IPC: B25J9/16 ,  G06T7/80

Abstract: A robot control system and a method for camera calibration verification is presented. The robot control system is configured to perform a first camera calibration, and to control a robot arm to move a verification symbol to a reference location. The robot control system further receives, from a camera, a reference image of the verification symbol, and determines a reference image coordinate for the verification symbol. The robot control system further controls the robot arm to move the verification symbol to the reference location again during an idle period, receives an additional image of the verification symbol, and determines a verification image coordinate. The robot control system determines a deviation parameter value based the reference image coordinate and the verification image coordinate, and whether the deviation parameter value exceeds a defined threshold, and performs a second camera calibration if the threshold is exceeded.