The invention discloses a pigment-removing cosmetic composition and the application thereof. The composition mainly comprises elm bark polysaccharide, rice fermentation product filtrate and active ingredients of traditional Chinese medicine extract, wherein the traditional Chinese medicine is composed of liquorice, salvia miltiorrhiza and radix angelicae. The pigment-removing cosmetic compositioncan achieve effects of reaching de-keratinization activity on pigment epithelial cells, promoting detachment of cells with produced pigments, inhibiting tyrosinase activity and reducing melanogenesis,wherein the elm bark polysaccharide has a strong reduction effect and is capable of effectively scavenging free radicals; the liquorice in the traditional Chinese medicine extract has strong absorption on ultraviolet rays and inhibits the tyrosinase activity, the salvia miltiorrhiza promotes skin blood circulation, and the radix angelicae has effects of whitening and protecting skin; the rice fermentation product filtrate has effects of promoting skin moisture absorption and lightening pigments and whitening, and when the composition is applied to preparation of cosmetics, the cosmetic can beendowed with an obvious whitening effect of removing the pigments.