A method of coding video data includes receiving one or more layers of video information. Each layer may include at least one picture. The method can include processing an indicator within at least one of a video parameter set (VPS), a sequence parameter set (SPS), or a picture parameter set (PPS) that indicates whether all direct reference layer pictures associated with the at least one of the video parameter set (VPS), the sequence parameter set (SPS), or the picture parameter set (PPS) are added to an inter-layer reference picture set. Based on the indicator, the method can further include refraining from further signaling inter-layer reference picture information in any video slice associated with the at least one of the video parameter set (VPS), the sequence parameter set (SPS), or the picture parameter set (PPS). Alternatively, based on the indicator, the method can further include adding to the inter-layer reference picture set all direct reference layer pictures for any video slice associated with the at least one of the video parameter set (VPS), the sequence parameter set (SPS), or the picture parameter set (PPS).