A method of controlling a portable device based on variation of a magnetic field by a detachable pen attached to the portable device and a portable device thereof is provided. The method includes detecting variation of a magnetic field due to at least one of an attaching, a detaching, and a rotation of the pen, determining whether there is a control operation corresponding to the variation of the magnetic field, and performing the control operation corresponding to the variation of the magnetic field when there is the control operation.