A stubble lean detection system for an agricultural harvester includes a controller having a memory and a processor. The controller is configured to receive a sensor signal indicative of an image of stubble within an agricultural field. The controller is also configured to determine whether an aggregate direction of lean of the stubble is forward, rearward, or mixed based on the image. Furthermore, the controller is configured to output an information signal to a user interface indicative of instructions to inform an operator of the aggregate direction of lean.