A color classification system and method for classifying, judging and/or identifying objects by color. An optical system utilizes light from an object to form an image on an imaging device. A plurality of optical band-pass filters each having a different pass band are provided between the object and the imaging device. The optical band-pass filters include a plurality of measurement filters for use in measuring a color of the object and a test filter for use in judging which one of the measurement filters should be applied to the object. A changeover section changes between the optical band-pass filters. A computing circuit performs at least one of a color classification and a color judgment based on a spectral characteristic of the imaged object through a statistical approach, and a control section causes the changeover section to select an effective one of the measurement filters based on a judgment by the computing circuit of a result sensed by the test filter. The plurality of optical band-pass filters are arranged in a plurality of filter sets each of which has at least one filter, and the changeover section includes a filter changeover mechanism for changing between the optical band-pass filters in one of the filter sets, and a filter-set changeover mechanism for changing between the plurality of filter sets.