A method for inter-color residual prediction using an improved scaling parameter coding is disclosed. In one embodiment, a scaling parameter is used for inter-color residual prediction between a first color component and a second color component. The scaling parameter is coded using context-based coding having the same number of context models as the longest binary code for the absolute values of the scaling parameter. For example, when the first scaling parameter is selected from a group consisting of 8, 4, 2, 1, 0, −1, −2 −4 and −8, the absolute values of the scaling parameter can be represented by binary codes with the longest binary code having 4 bits. The context-based coding for the absolute scaling parameter also uses 4 context models. In a second embodiment, the scaling parameter is coded using binary codes by assigning a shorter binary code to a larger non-zero scaling parameter.