Systems and methods for perceiving a scene around a mobile device include a sensor system capturing an image of and object list data for the scene around the mobile device. The image includes a point cloud with group(s) of points. An object list is generated from the object list data, and the object list indicates a location and a speed of each surface in a set of surfaces in the scene. A perception system generates a selected perception using the image and the object list. The selected perception is generated based on a comparison of first, second, and third perceptions within a hierarchy. The first perception includes object speed and object location. The second perception includes object classification and object tracking, and the third perception includes object type and object model. A motion planner generates a motion plan for the mobile device based on the selected perception.