A system and method of detecting drowsy driving that include a controller configured to calculate basic characteristics based on steering torque information and lateral displacement information for every first time and basic thresholds based on the basic characteristics for a second time. In addition, the controller calculates a current characteristic based on the steering torque information and the lateral displacement information for every third time and calculates current thresholds based on the current characteristic for a fourth time. The controller then compares the current thresholds and the current characteristics to determine whether drowsy driving is generated.