A system and method for actively selecting and labeling images for semantic segmentation are disclosed. A particular embodiment includes: receiving image data from an image generating device; performing semantic segmentation or other object detection on the received image data to identify and label objects in the image data and produce semantic label image data; determining the quality of the semantic label image data based on prediction probabilities associated with regions or portions of the image; and identifying a region or portion of the image for manual labeling if an associated prediction probability is below a pre-determined threshold.