An image decoding device includes: a decoding-side first generation unit that generates, for a target unit, a prediction sample of a first component; a decoding-side linear prediction unit that uses the sample of the first component and a prediction coefficient to generate a linear prediction sample of a second component; and a decoding-side second generation unit that uses the linear prediction sample of the second component to generate, for the target unit, a prediction sample of the second component. The decoding-side linear prediction unit uses a first coefficient as the prediction coefficient when the sample of the first component is equal to or below a threshold value, and uses a second coefficient which is different from the first coefficient as the prediction coefficient when the sample of the first component is greater than the threshold value. The threshold value is set on the basis of a parameter representing a distribution of or a change in a reference sample of the first component and/or the second component contained in a reference unit which is referenced for the target unit.