Color sorting system and method which are particularly suitable for sorting fruits and vegetables. The objects to be sorted are scanned with a color video camera, and the signals from the camera are digitized and utilized to address a look up table. The look up table is preloaded to provide reject data at those addresses for colors to be rejected. Several techniques for loading the look up table are disclosed. Then, on an online basis, the successive images address the look up table and the reject data is analyzed to drive appropriate reject apparatus. In one embodiment, the data from the look up table is applied to a spatial filter, and objects are rejected only if they have a certain number or sequence of unacceptable colors.