The invention relates to a power data de-duplication method used on a big data platform. The overall framework for deleting repeated data is in a distributed mode; the distributed overall framework distributes the de-duplication process to multiple nodes for execution so as to avoid system bottlenecks and single-point faults. Hash calculation is conducted on the data content before duplication judgment of the data, and the file content with a variable length is converted into a fingerprint with a fixed length so that the duplication judgment efficiency can be greatly improved. Two algorithms are used in Hash calculation and include the MD5 algorithm and the SHA-1 algorithm. Two functions are used for calculation, so that the possibility that some functions have data conflicts is reduced, that is, different data contents can correspond to the same data hash value.