A portable device is disclosed. A method of controlling a portable device, comprising the steps of capturing an image in front of the portable device, detecting a marker object from the image, displaying a virtual image corresponding to the marker object based on a location of the marker object, and terminating a display of the virtual image corresponding to the marker object, when the detecting of the marker object is terminated, terminate a display of the virtual image based on a first terminate mode if the gaze location of the user is detected at a first location, and terminate the display of the virtual image based on a second terminate mode if the gaze location of the user is detected at a second location.