A first subset of pixels of an image may be labeled with an object identifier based on user interactions with the image. Pixel data representing the pixels of the image may be passed through an embedding neural network model to generate pixel embedding vectors. A prototype embedding vector associated with the object identifier may be generated based pixel embedding vectors corresponding to the first subset of pixels. For each pixel of a second subset of pixels of the image, a probability that the pixel should be labeled with the object identifier may be determined based on the prototype embedding vector and pixel embedding vectors corresponding to the second subset of pixels. Pixels of the second subset of pixels may be labeled with the object identifier based on the determined probabilities, and the pixels in the image may be segmented based on the pixels labeled with the object identifier.