An external controller has greater control over control circuitry on a memory die in a non-volatile storage system. The external controller can issue a manual suspend command on a communication path which is constantly monitored by the control circuitry. In response, the control circuitry suspends a task immediately, with essentially no delay, or at a next acceptable point in the task. The external controller similarly has the ability to issue a manual resume command, which can be provided on the communication path when that path has a ready status. The control circuitry can also automatically suspend and resume a task. The external controller can cause a task to be suspended by issuing an illegal read command. The external controller can cause a suspended program task to be aborted by issuing a new program command.