Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating descriptive text for images. In one aspect, a method includes identifying a set of seed descriptors for an image in a document that is hosted on a website. For each seed descriptor, structure information is generated that specifies a structure of the document with respect to the image and the seed descriptor. One or more templates are generated for each seed descriptor using the structure information for the seed descriptor. Each template can include image location information, document structure information, image feature information, and a generative rule that generates descriptive text for other images in other documents. Descriptive text for other images is generated using the templates and the other documents. The descriptive text is associated with the images.