An apparatus and method for performing a compression operation in a fast message hash algorithm, which receive a 512-bit message and 512-bit chaining variable data, repeatedly calculate a 128-bit register-based step function, and then produce updated 512-bit chaining variable data. For this, the apparatus for performing a compression operation in a hash algorithm includes a message extension unit for receiving a message and generating a plurality of extended messages. A chaining variable initial conversion unit receives chaining variable data and converts the chaining variable data into initial state data. A step function operation unit repeatedly calculates a step function based on the initial state data and the plurality of extended messages and produces final state data. A chaining variable final conversion unit generates updated chaining variable data from the chaining variable data using the final state data, and outputs the updated chaining variable data.