An apparatus and method for generating and consuming a three-dimensional (3D) data format to generate a realistic panoramic image are provided. The apparatus may include an image preprocessing unit to search for a matching point between images captured by a plurality of cameras, and to extract, as image information, at least one of a depth value, a texture value and object division information from each of the captured images, an image information structuring unit to structure 3D data to use the extracted image information to generate a realistic image, a 3D data format storage unit to store format information of the structured 3D data in a database (DB), realistic image generating unit to generate a realistic panoramic image using the stored format information of the 3D data, and a realistic image rendering unit to perform rendering on the generated realistic panoramic image.