A computer-implemented method for improving generalization in training deep neural networks in online settings. The method comprises a general learning paradigm for sequential data that is referred to as Learn, Unlearn, RElearn (LURE), a dynamic re-initialization method to address the above-mentioned larger problem of generalization of parameterized networks on sequential data by selectively retaining the task-specific connections through the important criteria and re-randomizing the less important parameters at each mega batch of training. The proposed method of selectively forgetting is crucial for retaining previous information all the while improving generalization to unseen samples.