Disclosed herein is an image management system including an image output apparatus and a server. The image output apparatus includes an obtaining unit to obtain user information when the image data is output, a history storage unit to accumulate and store the user information which is obtained by the obtaining unit and the image data which is output as history information, a history transfer unit to transfer the history information which is stored in the history storage unit to the server, and a deletion unit to delete the history information which is transferred by the history transfer unit from the history storage unit. The server includes a receiving unit to receive the history information which is transferred from the image output apparatus, and a history management unit to accumulate and store the history information which is received by the receiving unit.