Systems and methods are described for rendering a 3D synthetic scene. A display client receives point cloud samples of the scene, where the point cloud samples include a point location, one or more viewing directions, and color information for each of the viewing directions. The point cloud samples may be generated by a server using ray tracing. The display client combines information from the point cloud samples with a locally generated rendering of at least a portion of the scene to generate a combined rendering of the scene, and the client displays the combined rendering. The number of point cloud samples may be adjusted adaptively based on performance metrics at the client.