A terminal operation system includes a plurality of terminals connected via a network, wherein an operated terminal can be operated by an operating terminal through an input device, such as a pointer input part or a key input part, connected to the operating terminal. The system controls an appropriate assignment of the operation authority and a smooth transferring of the operation authority among the operating terminals. The operating terminal operates the operated terminal based on a certain level of an operation authority which is layered into a plurality of levels according to the possible function to perform on the operated terminal such as key code input, mouse operation, pointer moving and so on.