A system for calibrating tool depth as an agricultural implement is moved across a field may include an implement frame and a ground-engaging tool supported on the implement frame, with the ground-engaging tool configured to penetrate soil present within the field to a penetration depth. Additionally, the system may also include a frame position sensor configured to capture data indicative of a distance between the implement frame and a surface of the field. Furthermore, a controller of the disclosed system may be configured to determine a penetration depth value of the ground-engaging tool based on a received input. Moreover, the controller may be configured to determine a correction factor based on the data captured by the frame position sensor. In addition, the controller may be configured to adjust the determined penetration depth value based on the determined correction factor to calibrate the penetration depth value.