A conference system in which a server and a plurality of user terminals are communicably connected is provided. The server has a function of setting permission and prohibition of output of speech voice signals of one or more other users for each user who is a user of each of the user terminals. When a specific user among a plurality of users participating in a conference speaks, the server, based on the setting, transmits the speech voice signals to user terminals of permitted users who are permitted to output the speech voice signals with respect to the specific user, and omits transmission of the speech voice signals of user terminals of prohibited users who are prohibited from outputting the speech voice signals with respect to the specific user.