What is disclosed is a system and method for increasing the accuracy of physiological signals obtained from video of a subject being monitored for a desired physiological function. In one embodiment, image frames of a video are received. Successive batches of image frames are processed. For each batch, pixels associated with an exposed body region of the subject are isolated and processed to obtain a time-series signal. If movement occurred during capture of these image frames that is below a pre-defined threshold level then parameters of a predictive model are updated using this batch's time-series signal. Otherwise, the last updated predictive model is used to generate a predicted time-series signal for this batch. The time-series signal is fused with the predicted time-series signal to obtain a fused time-series signal. The time-series signal for each batch is processed to obtain a physiological signal for the subject corresponding to the physiological function.