A method and device are disclosed from the perspective of a first device handling radio link failure. In one method, the method includes the first device performing or establishing sidelink unicast communication with a second device. The method also includes the first device performing or establishing sidelink unicast communication with a third device. The method also includes the first device detecting a radio link failure associated with the second device. The method further includes the first device performing a Sidelink MAC reset associated with the second device in response to the radio link failure, and not performing a Sidelink MAC reset associated with the third device.