Method and apparatus for encoding, decoding and rendering 3D media content are provided. An apparatus for rendering three-dimensional (3D) media content includes a communication interface configured to receive a multimedia stream, and one or more processors operably coupled to the communication interface, the one or more processors configured to parse the multimedia stream into 2D video bitstreams including geometry frames and texture frames, 2D to 3D conversion metadata for rendering 3D points from 2D frames, and scene description metadata describing 6 degree of freedom (6DoF) relationships among objects in a 6DoF scene, decode the 2D video streams including geometry data and texture data to generate 2D pixel data, covert the 2D pixel data into 3D voxel data using the 2D to 3D conversion metadata; and generate the 6DoF scene from 3D voxel data using the scene description metadata.