Methods, systems, and apparatus, including computer programs encoded on computer storage media, for sensor calibration with environment map. In some implementations, a three-dimensional surfel representation of a real-world environment is obtained. One or more surfels of the surfel representation having a particular classification of the different classifications are selected. Input sensor data from one or more sensors installed on an autonomous or semi-autonomous vehicle are received. The input sensor data is compared to the surfel representation to identify one or more differences between the observation and the surfel representation. At least one sensor of the one or more sensors is calibrated using the one or more differences between the observation and the surfel representation.