A method for training a system for detecting multi-class objects in an image or a video sequence is described. A common ensemble of weak classifiers for a set of object classes is identified. For each object class, a separate weighting scheme is adapted for the ensemble of weak classifiers. A method for detecting objects of multiple classes in an image or a video sequence is also disclosed. Each class is assigned a detector that is implemented by a weighted combination of weak classifiers such that all of the detectors are based on a common ensemble of weak classifiers. Then weights are individually set for each class.