The present disclosure relates to systems, methods, and tools for tool tracking using image-derived data from one or more tool-located references features. In some embodiments, a medical system includes a tool having a distal end that is insertable into a patient body, an image capture device insertable into the patient body so that the image capture device captures an image of at least a portion of a two-dimensional marker at least partially surrounding a portion of the tool, and a processor coupled to the image capture device and configured to determine a pose of the tool by processing the image.