Devices, methods, and non-transitory program storage devices are disclosed herein to provide improved region of interest (ROI) tracking, image framing, and distortion correction for wide field of view (FOV) video image streams. The techniques disclosed may be configured, such that the image framing decisions made over the course of a captured wide FOV video image stream are able to be panned smoothly and seamlessly transition between framing a narrower portion of a wide angle camera's FOV to framing a wider portion of the wide angle camera's FOV (or vice versa), e.g., depending on the composition and movement of relevant subjects in the captured scene. The techniques disclosed herein may also be used to perform any distortion correction desired on the framed image portions in the output video image stream, e.g., based on the image portions' FOVs and/or the locations of the portions within the original wide FOV video image stream.