A head mounted display device including a processor configured to compute a rendered rendering surface of a predicted scene having a predicted user viewpoint, the predicted user viewpoint being a prediction of a viewpoint that a user will have at a point in time that was predicted for the user of the head mounted display device prior to the point in time, receive, from the user input device, a subsequent user navigation input near the point in time in the stream of user input, determine an actual user viewpoint based on the subsequent user navigation input, determine a user viewpoint misprediction based on the predicted user viewpoint and the actual user viewpoint, and reconstruct a viewport for the actual user viewpoint from the rendered rendering surface.