The invention relates to a die holder structure capable of reducing magnet breaking. The die holder structure is characterized in that: an upper male die is fixedly installed on an upper male die plate, a lower male die is fixedly installed on a lower male die plate, the upper end of the lower male die is tightly sleeved inside a female die, the female die is fixedly installed in a female die plate, the upper end of a core rod is tightly sleeved inside the lower male die, and a die cavity is formed among the female die, the lower male die and the core rod; and the lower end of the core rod is fixedly connected with a cylinder and the core rod can move upwards and downwards under the control of the cylinder. The die holder structure provided by the invention works in such cooperative ways that: a demolding device of a press drives the female die plate to move downwards, the core rod is kept to be stationary under the control of the cylinder, and the external surface of a magnet green compact is completely separated from the female die; and then the core rod moves downwards under the control of the cylinder, and the internal surface of the magnet green compact is completely separated from the core rod, thus demolding is completed. In the invention, the demolding process is completed by two steps, thus the magnet green compact breaking caused because the internal and external surfaces undergo friction force at the same time is avoided, and the problems existing in the prior art are effectively solved.