In an in-vehicle handsfree telephone system, an in-vehicle handsfree device is capable of transferring a voice signal to a portable terminal in a passenger compartment of a vehicle. An operating state detection unit detects an operating state of the vehicle. The handsfree device is responsive to a user operation and selectively operable in one of a handsfree mode in which talking on the portable terminal through the handsfree device is allowed by the transfer of the voice signal and a portable terminal stand-alone mode in which talking on the portable terminal in isolation is allowed. The handsfree device is configured to have a switching restriction function in which the handsfree device restricts switching between the handsfree mode and the portable terminal stand-alone mode in accordance with a detected operating state of the vehicle.