In a voice data communication system comprising a switchboard for switching lines, a network, interworking units for interworking a protocol on the network with a protocol for a line switching, and IP extension terminals connected to the interworking units through the network, a switchboard notifies a communication between interworking units to the interworking units when recognizing the communication between the interworking units, which are provided corresponding to each of IP extension terminals, and establish a relay route to the interworking unit according to a notification from the switchboard and establish a voice data packet route to a route between the IP extension terminals through the network by using the relay route in response to a route establishment request from the IP extension terminals accommodated, or when the interworking unit is a single and the notification is received from the switchboard, establish a voice data packet route to a route between the IP extension terminals through the network according to a route establishment request from the IP extension terminal accommodated.