A processor comprises a first memory to store data elements that are encoded according to a floating point format including a sign field, an exponent field, and a significand field; and a compression engine comprising circuitry, the compression engine to generate a compressed data block that is to include a tag type per data element, wherein responsive to a determination that a first data element includes a value in its exponent field that does not match a value of any entry in a dictionary, a first tag type and an uncompressed value of the data element are included in the compressed data block; and responsive to a determination that a second data element includes a value in its exponent field that matches a value of a first entry in the dictionary, a second tag type and a compressed value of the data element are included in the compressed data block.