An example image feature extraction system comprises an encoder neural network having a first set of layers and a decoder neural network having a second set of layers and a third set of layers. The encoder neural network receives an input image, processes the input image through the first set of layers, and computes an encoded feature map based on the input image. The decoder neural network receives the encoded feature map, processes the encoded feature map through the second set of layers to compute a keypoint score map, and processes the encoded feature map through at least a portion of the third set of layers to compute a feature description map.