Techniques and systems are described for encoding 360-degree video data using the planes of a truncated square pyramid to map the 360-degree data for different fields of view. 360-degree video data can include multiple frames, where each frame includes spherical video data. In various implementations, a video coding system can select a field of view for the video data, and determine an offset from the center of the spherical video data that corresponds to the field of view. Using the offset, the system can determine a projection of the spherical video data onto the planes of the truncated square pyramid, where the base plane represents a front view and the top plane represents a back view. The system can then map the video data according to the projection such that each plane of the truncated square pyramid includes a portion of the spherical video data.