Disclosed are method and system for detecting harmful files executed by a virtual stack machine. An example method includes: analyzing a file executable on the virtual stack machine to identify both parameters of a file section of the file and parameters of a function of the virtual stack machine when executing the file; identifying, in a database, at least one cluster of safe files based on the identified parameters of the file section of the file and the identified parameters of the virtual stack machine; creating, using at least one clustering rule, a data cluster based on the identified at least one cluster of safe files; calculating at least one checksum of the created data cluster; and determining that the file executable on the virtual stack machine is harmful if the computed at least one checksum matches a checksum in a database of checksums of harmful files.