The invention belongs to the technical field of polysilicon purification by a physical metallurgy technology, in particular to a method for removing phosphorus and boron of impurities in polysilicon by an electron beam smelting technology. In the method, two electron guns are used for emitting electron beams to respectively smelt the polysilicon; meanwhile, the phosphorus and the boron in the polysilicon are removed by a dual technology; the phosphorus in the polysilicon is removed at first, the polysilicon with low phosphorus is further smelted and evaporated to remove the boron; and the polysilicon with low phosphorus and boron, which is evaporated, on a depositing plate is collected. The adopted device comprises a shell and a vacuum chamber, wherein the shell comprises a vacuum cover and a vacuum barrel; an inner cavity of the vacuum barrel is the vacuum chamber; and the vacuum chamber comprises a left cavity and a right cavity which are partitioned by an isolating plate. The invention effectively improves the purity of the polysilicon, reaches the use requirement of solar energy-grade silicon and has good purifying effect, stable technology and high efficiency.