Systems and techniques are described for determining image handle locations. An image is provided to a neural network as input, and the neural network translates the input image to an output image that includes clusters of pixels against a background that have intensities greater than an intensity of the background and that indicate candidate handle locations. Intensities of clusters of pixels in an output image are compared to a threshold intensity level to determine a set of the clusters of pixels satisfying an intensity constraint. The threshold intensity level can be user-selectable, so that a user can control a density of handles. A handle location for each cluster of the set of clusters is determined from a centroid of each cluster. Handle locations include a coordinate for the handle location and an attribute classifying a degree of freedom for a handle at the handle location.