Described is a system for online vehicle recognition in an autonomous driving environment. Using a learning network comprising an unsupervised learning component and a supervised learning component, images of moving vehicles extracted from videos captured in the autonomous driving environment are learned and classified. Vehicle feature data is extracted from input moving vehicle images. The extracted vehicle feature data is clustered into different vehicle classes using the unsupervised learning component. Vehicle class labels for the different vehicle classes are generated using the supervised learning component. Based on a vehicle class label for a moving vehicle in the autonomous driving environment, the system selects an action to be performed by the autonomous vehicle, and causes the selected action to be performed by the autonomous vehicle in the autonomous driving environment.