A user terminal device capable of attaching and detaching a pen is provided. The device includes a detector configured to detect a user manipulation regarding a screen, and a controller configured to change a layout of the screen to correspond to a pen use mode in response to the pen being removed from the user terminal device, perform a control operation corresponding to the detected user manipulation in response to the user manipulation using the pen being detected, and restore the layout of the screen to its original state in response to the pen being remounted on the user terminal apparatus.