Systems and methods are disclosed for road scene understanding of vehicles in traffic by capturing images of traffic with a camera coupled to a vehicle; generating a continuous model of occlusions with a continuous occlusion mode for traffic participants to enhance point track association accuracy without distinguishing between moving and static objects; applying the continuous occlusion model to handle visibility constraints in object tracks; and combining point track association and soft object track modeling to improve 3D localization accuracy.