The technology provides a sign detection and classification methodology. A unified pipeline approach incorporates generic sign detection with a robust parallel classification strategy. Sensor information such as camera imagery and lidar depth, intensity and height (elevation) information are applied to a sign detector module. This enables the system to detect the presence of a sign in a vehicle's externa environment. A modular classification approach is applied to the detected sign. This includes selective application of one or more trained machine learning classifiers, as well as a text and symbol detector. Annotations help to tie the classification information together and to address any conflicts with different the outputs from different classifiers. Identification of where the sign is in the vehicle's surrounding environment can provide contextual details. Identified signage can be associated with other objects in the vehicle's driving environment, which can be used to aid the vehicle in autonomous driving.