Provided are a device and method of controlling a wearable device. The device may include a communicator configured to communicate with one or more wearable devices that are worn on a user's body and obtain physical information of the user; and a controller configured to determine wearing positions where the one or more wearable devices are worn on the user's body, to determine a type of activity (e.g., exercise) performed by the user, and to determine, as a main controlled-device based on the wearing positions, at least one wearable device that corresponds to the activity type of the user and is from among the one or more wearable devices worn on the user's body, wherein, via the communicator, the controller controls, from among the one or more wearable devices worn on the user's body, the main controlled-device differently from wearable devices excluding the main controlled-device.