A latency masking system for use in an autonomous vehicle (AV) system includes a sensors module providing sensor data from a plurality of sensors. The sensor data includes image frames provided by a vehicle camera and vehicle motion data. A wireless transceiver transmits the sensor data to a remote server associated with a network infrastructure and receives remote state information derived from the sensor data. An on-board function module receives the sensor data from the sensors module and generates local state information. A state fusion and prediction module receives the remote station information and the local state information and updates the local state information with the remote state information. The state fusion and prediction module uses checkpoints in a state history data structure to update the local state information with the remote state information.