Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for labeling images. In one aspect, a method includes automatically identifying an object in an image using a deep model-based and data-driven hybrid architecture.