A user terminal device is provided, which includes a sensor configured to sense a user operation with respect to a display; and a controller configured to operate in a first interaction mode in which an operation that corresponds to a touch point of a pen is performed if a pen touch with respect to the display is sensed by the sensor, to switch over to a second interaction mode if the controller determines that the pen is put on the display, and to perform an operation that corresponds to a line on the display on which the pen is put.