A video decoder may be configured to receive, in response to receiving a first syntax element indicating that reference picture list information is included in a picture header syntax structure, a second syntax element in the picture header syntax structure indicating whether a collocated picture used for temporal motion vector prediction is to be derived from a first reference picture list or a second reference picture list; receive a slice of the video data that refers to the picture header syntax structure; and in response to the slice being a P slice, set a value for a third syntax element associated with the slice to a first value for the third syntax element, with the first value for the third syntax element indicating that the collocated picture used for temporal motion vector prediction is to be derived from the first reference picture list.