An optical flow estimation process based on a spatio-temporal model with varying coefficients multiplying a set of basis functions at each pixel. The benefit of over-parameterization becomes evident in the smoothness term, which instead of directly penalizing for changes in the optic flow, accumulates a cost of deviating from the assumed optic flow model. The optical flow field is represented by a general space-time model comprising a selected set of basis functions. The optical flow parameters are computed at each pixel in terms of coefficients of the basis functions. The model is thus highly over-parameterized, and regularization is applied at the level of the coefficients, rather than the model itself. As a result, the actual optical flow in the group of images is represented more accurately than in methods that are known in the art.