The present invention relates to an image data reconstructing apparatus for reconstructing an original image by dividing image data into a plurality of blocks each comprising N×N picture elements, performing two-dimensional orthogonal transform on each of the blocks, quantizing and encoding the result of the two-dimensional orthogonal transform, decoding the encoded result, dequantizing and two-dimensional inverse orthogonal transforming the above result, wherein an operation associated with the multiplication is skipped if a multiplication by zero is detected when two-dimensional inverse orthogonal transform is performed. Before performing dequantization, two block volume of buffer is provided before or after performing two-dimensional inverse orthogonal transform for executing a pipeline process. Furthermore, two-dimensional inverse orthogonal transform is performed by repeating two times a process of one-dimensional inverse orthogonal transform and matrix transposition, thus the matrix transposition being performed as the result of one-dimensional inverse orthogonal transform which is stored in buffer.