Systems, methods, and apparatuses described herein are directed to performing segmentation on voxels representing three-dimensional data to identify static and dynamic objects. LIDAR data may be captured by a perception system for an autonomous vehicle and represented in a voxel space. Operations may include determining a drivable surface by parsing individual voxels to determine an orientation of a surface normal of a planar approximation of the voxelized data relative to a reference direction. Clustering techniques can be used to grow a ground plane including a plurality of locally flat voxels. Ground plane data can be set aside from the voxel space, and the remaining voxels can be clustered to determine objects. Voxel data can be analyzed over time to determine dynamic objects. Segmentation information associated with ground voxels, static object, and dynamic objects can be provided to a tracker and/or planner in conjunction with operating the autonomous vehicle.