A sorting unit sorts a plurality of data sets of HP component having been processed by a decoding unit, selectively employing one of a first table corresponding to a first orientation of prediction and a second table corresponding to a second orientation of prediction in accordance with an orientation of prediction of HP component. The sorting unit includes an inverse prediction unit performing inverse prediction on data of LP component inputted from the decoding unit, a processing unit obtaining an orientation of prediction of HP component, based on the data of LP component after inverse prediction by the inverse prediction unit, and a selecting unit selecting one of the first and second tables, based on the orientation of prediction of HP component obtained by the processing unit.