In one aspect, a system for controlling the operation of a seed-planting implement may include a furrow-forming tool configured to form a furrow in soil present within a field. Furthermore, the system may include a sensor configured to capture data indicative of a topographical profile of the soil within the field. Additionally, a controller of the disclosed system may be configured to identify a topographical feature within the field based on the data received from the sensor. Furthermore, the controller may be configured to determine a position of the furrow-forming tool relative to the identified topographical feature. Additionally, the controller may be configured to initiate a control action to adjust the position of the furrow-forming tool when it is determined that the relative position between the furrow-forming tool and the identified topographical feature is offset from a predetermined positional relationship defined for the furrow-forming tool.