Techniques are described for 3D lookup table based color gamut scalability in multi-layer video coding. A lower layer of video data may include color data in a first color gamut and a higher layer of the video data may include color data in a second color gamut. To generate inter-layer reference pictures, a video coder performs color prediction to convert the color data of a reference picture in the first color gamut to the second color gamut. The video coder may perform color prediction using a 3D lookup table. According to the techniques, the video coder generates at least one 3D lookup table having a size that is different for a luma component than for a first chroma component and a second chroma component. In this way, the total size of the table may be reduced while maintaining good coding performance.