Systems and methods of correcting for signal drift in optical image stabilization of a portable device. An accelerometer is used to obtain an acceleration signal corresponding to acceleration of the portable device. A gravity component is removed from the accelerometer signal to obtain a proper acceleration signal. A double integration is performed on the proper acceleration signal to obtain a translation signal. A drift component of the translation signal is estimated. The estimated drift component is used to generate a drift correction signal. A phase compensation signal is determined based on a plurality of phase responses of a plurality of components of the optical image stabilization system. The drift correction signal and the phase compensation signal are applied to an actuator controller which controls movement of a lens within the portable device as part of the process for optical image stabilization.