Provided is a hand-held type user interface device for providing force feedback to a user according to an interaction with a virtual object or a user at a remote place, which remarkably increases the sense of reality in the interaction with the virtual object without limiting an action of the user by sensing the three-dimensional (3D) position and direction of the device and diversifying a force feedback provided to the user according to an action of the user holding the device.