One or more chat servers receives voice signals and pose (location and orientation) signals from devices such as VR headsets associated with respective chat participants. For each participant, the server renders a single stream representing the voices of the other participants, with the voice data in each stream being modified to account for the orientation of the head of the receiving participant. The server sends the streams to the participants for whom the streams are tailored. The voice information representing the chat of the other participants in a stream intended for a particular participant can also be modified to account for the distances between participants and orientations of speakers' heads relative to the particular participant for whom the stream is tailored.