A process for improving a purity and/or whiteness degree of a talc powder is disclosed. A talc powder with a particle size of about 0.8 mm or less containing impurities such as asbestos, etc. is treated with a phosphoric acid agent under heating at from 70.degree. to 250.degree. C. to remove impurities such as asbestos, etc. by utilizing the difference in the rate of dissolution as between the talc and the impurities. The phosphoric acid agent is one selected from the group consisting of phosphoric acid, pyrophosphoric acid, sodium phosphate, ammonium phosphate, calcium phosphate, sodium tripolyphosphate, sodium pyrophosphate, sodium hexametaphosphate and mixtures thereof. Thus, asbestos, which is considered to be carcinogenic, can be effectively removed from talc.