In one embodiment, a lane departure detection system detects at a first point in time that a wheel of an ADV rolls onto a lane curb disposed on an edge of a lane in which the ADV is moving. The system detects at a second point in time that the wheel of the ADV rolls off the lane curb of the lane. The system calculates an angle between a moving direction of the ADV and a lane direction of the lane based on the time difference between the first point in time and the second point in time in view of a current speed of the ADV. The system then generates a control command based on the angle to adjust the moving direction of the ADV in order to prevent the ADV from further drifting off the lane direction of the lane.