A detection system detects malfunctions in an autonomous farming vehicle during an autonomous routine using one or more models and data from sensors coupled to the autonomous farming vehicle. The models may include machine-learned models trained on the sensor data and configured to identify objects indicative of an operational or malfunctioning component within a tilling assembly such as a tilling shank or sweep. Additionally, a machine-learned model may be trained on sensor data to detect whether debris has plugged the tilling assembly of the autonomous farming vehicle. In response to detecting a malfunction or a plug, the detection system may modify the autonomous routine (e.g., pausing operation) or provide information for the malfunction to be addressed (e.g., the likely location of a malfunctioning sweep that has detached from the tilling assembly).