There is described a computing device and method in a digital medium environment for custom auto tagging of multiple objects. The computing device includes an object detection network and multiple image classification networks. An image is received at the object detection network and includes multiple visual objects. First feature maps are applied to the image at the object detection network and generate object regions associated with the visual objects. The object regions are assigned to the multiple image classification networks, and each image classification network is assigned to a particular object region. The second feature maps are applied to each object region at each image classification network, and each image classification network outputs one or more classes associated with a visual object corresponding to each object region.