A neural network is trained for use in a substrate thickness measurement system by obtaining ground truth thickness measurements of a top layer of a calibration substrate at a plurality of locations, each location at a defined position for a die being fabricated on the substrate. A plurality of color images of the calibration substrate are obtained, each color image corresponding to a region for a die being fabricated on the substrate. A neural network is trained to convert color images of die regions from an in-line substrate imager to thickness measurements for the top layer in the die region. The training is performed using training data that includes the plurality of color images and ground truth thickness measurements with each respective color image paired with a ground truth thickness measurement for the die region associated with the respective color image.