A virtual reality (VR) audio rendering system and method include spatializing microphone-captured real-world sounds according to a VR setting. In a game streaming system, when a player speaks through a microphone, the voice is processed by geometrical acoustic (GA) simulation configured for a virtual scene, and thereby spatialized audio effects specific to the scene are added. The GA simulation may include generating an impulse response using sound propagation simulation and dynamic HRTF-based listener directivity. When the GA-processed voice of the player is played, the local player or other fellow players can hear it as if the sound travels in the scenery and according to the geometries in the virtual scene. This mechanism can advantageously place the players' chatting in the same virtual world like built-in game audio, thereby advantageously providing enhanced immersive VR experience to users.