In implementations of systems for performing multiple segmentation tasks, a computing device implements a segment system to receive input data describing a digital image depicting an object. The segment system computes per-pixel embeddings for the digital image using a pixel decoder of a machine learning model. Output embeddings are generated using a transformer decoder of the machine learning model based on the per-pixel embeddings for the digital image, input embeddings for a first segmentation task and input embeddings for a second segmentation task. The segment system outputs a first digital image and a second digital image. The first digital image depicts the object segmented based on the first segmentation task and the second digital image depicts the object segmented based on the second segmentation task.