Methods, systems, and apparatus, including computer programs encoded on computer storage media, for labeling point clouds using distance field data. One of the methods includes obtaining a point cloud characterizing a region of the environment, the point cloud comprising a plurality of points; obtaining distance field data specifying, for each of a plurality of locations in the region of the environment, a distance from the location to a nearest static object in the environment; determining, using the distance field data and for each of the plurality of points, a respective distance from the point to a nearest static object in the environment to the point; and identifying, based on the respective distances for the plurality of points in the point cloud, one or more of the points as candidate dynamic object points that are likely to be measurements of a dynamic object in the environment.