A permanent magnet material is prepared by covering an anisotropic sintered magnet body of formula: R1x(Fe1-yCoy)100-x-z-aBzMa wherein R1 is a rare earth element, M is Al, Cu or the like, with a powder comprising an oxide of R2, a fluoride of R3 or an oxyfluoride of R4 wherein R2, R3, and R4 are rare earth elements, and having an average particle size up to 100 μm, heat treating the powder-covered magnet body in a hydrogen gas-containing atmosphere for inducing disproportionation reaction on R12Fe14B compound, and continuing heat treatment at a reduced hydrogen gas partial pressure for inducing recombination reaction to said compound, thereby finely dividing said compound phase to a crystal grain size up to 1 μm, and for effecting absorption treatment, thereby causing R2, R3 or R4 to be absorbed in the magnet body.