A system includes a neural network organized into layers corresponding to stages of inferences. The neural network includes a common portion, a first portion, and a second portion. The first portion includes a first set of layers dedicated to performing a first inference task on an input data. The second portion includes a second set of layers dedicated to performing a second inference task on the same input data. The common portion includes a third set of layers, which may include an input layer to the neural network, that are used in the performance of both the first and second inference tasks. The system may receive an input data and perform both inference tasks on the input data in a single pass. During training, a training sample with annotations for both inference tasks may be used to train the neural network in a single pass.