A method of encoding a three-dimensional (3D) image including a point cloud includes grouping a plurality of points included in the point cloud into at least one segment; generating patches by projecting the points included in the segment onto a predetermined plane in a first direction or a second direction; generating two-dimensional (2D) images by packing the patches; and generating and outputting a bitstream including information about a direction in which each point is projected to generate the patches and information about the 2D images.