The invention provides a harmful substance insolubilizing material which can effectively insolubilize the harmful substance in a short time. The harmful substance insolubilizing material provided by the invention includes a hydroxide dolomite and a phosphoric acid compound. The harmful substance is more than one substance selected from the second special harmful substance from drainage, soil, incineration ash or coal ash and antimony. The molar ratio P/Ca of phosphorus in the phosphoric acid compound to 1 mol calcium in the hydroxide dolomite is 0.1-0.7.