Estimation of a 3D layout of roads and paths traveled by pedestrians is achieved by observing the pedestrians and estimating road parameters from the pedestrian's size and position in a sequence of video frames. The system includes a foreground object detection unit to analyze video frames of a 3D scene and detect objects and object positions in video frames, an object scale prediction unit to estimate 3D transformation parameters for the objects and to predict heights of the objects based at least in part on the parameters, and a road map detection unit to estimate road boundaries of the 3D scene using the object positions to generate the road map.