Systems and methods are disclosed for collecting driving data from simulated autonomous driving vehicle (ADV) driving sessions and real-world ADV driving sessions. The driving data is processed to exclude manual (human) driving data and to exclude data corresponding to the ADV being stationary (not driving). Data can further be filtered based on driving direction: forward or reverse driving. Driving data records are time stamped. The driving data can be aligned according to the timestamp, then a standardized set of metrics is generated from the collected, filtered, and time-aligned data. The standardized set of metrics are used to grade the performance the control system of the ADV, and to generate an updated ADV controller, based on the standardized set of metrics.