Compensating for latency in a streaming virtual reality environment is disclosed. A computing device receives from a head-mounted display device (HMDD) having a field of view (FOV), first pose data that identifies an orientation of the HMDD worn by a participant viewing a virtual world via the HMDD. The computing device renders, based at least in part on the first pose data and a first oversized FOV that is greater than the FOV, oversized imagery that depicts a portion of the virtual world within the FOV of the HMDD and a portion of the virtual world outside of the FOV of the HMDD. The computing device communicates the oversized imagery to the HMDD.