A system and a method of sorting scrap particles includes imaging a moving conveyor containing scrap particles using a vision system to create an image. A computer analyzes the image as a matrix of cells, identifies cells in the matrix containing a particle, and calculates a color input for the particle from a color model by determining color components for each cell associated with the particle. A light beam is directed to the particle on the conveyor downstream of the vision system, and at least one emitted band of light from the particle is isolated and detected at a selected frequency band to provide spectral data for the particle. The computer generates a data vector for the particle containing the color input and the spectral data, and classifies the particle into one of at least two classifications of a material as a function of the vector.