The vacuum smelting process of extracting selenium includes two steps, including the selenium material forming and drying step and vacuum furnace extraction step. In the selenium material forming and drying step, selenium material is mixed with adhesive, and the mixture is produced into pellet or block and dried to water content less than 1 %. In the vacuum furnace extraction step, the material pellet or block is loaded into material boat and the material boat is set inside vacuum furnace for vacuum smelting. Selenium vapor generated inside the furnace is trapped in selenium trap and made to flow via heat insulating pipeline into selenium pot and the slag is discharged from the boat. The said process has high recovering rate and low production cost.