A method in a vehicle for performing multiple on-board sensing tasks concurrently in the same network using deep learning algorithms is provided. The method includes receiving vision sensor data from a sensor on the vehicle, determining a set of features from the vision sensor data using a plurality of feature layers in a convolutional neural network, and concurrently estimating, using the convolutional neural network, bounding boxes for detected objects, free-space boundaries, and object poses for detected objects from the set of features determined by the plurality of feature layers. The neural network may include a plurality of free-space estimation layers configured to determine the boundaries of free-space in the vision sensor data, a plurality of object detection layers configured to detect objects in the image and to estimate bounding boxes that surround the detected objects, and a plurality of object pose detection layers configured to estimate the direction of each object.