Methods and system are provided for training and using a model to determine states of lanes of interest. For instance, image data including an image and an associated label identifying at least one traffic light, a state of the at least one traffic light, and a lane controlled by the at least one traffic light are received and used to train the model such that the model is configured to, in response to receiving an image and a lane of interest included in the image, output a lane state for the lane of interest. This model is then used by a vehicle in order to determine a state of a lane of interest. This state is then used to control the vehicle in an autonomous driving mode based on the state of the lane of interest.