Described herein is a system and method for adjusting the orientation of captured video utilizing the data received from a built-in inertial measurement unit such as an accelerometer. During video capture, the device may be held in a position that is not fully vertical or horizontal, and thus not match the true orientation of a scene. This can cause the captured video to appear rotated during playback at the same angle of rotation as the video capture device. The described system can adjust the playback orientation of the rotated video by utilizing sensor data captured during the same time as the video was captured.