Systems, methods, and devices for optimizing phase detection autofocus (PDAF) processing are provided. One aspect provides an apparatus comprising: an image sensor configured to capture image data of a scene; a buffer; and a processor. The processor may be configured to store the image data in the buffer as a current frame and divide the current frame into a plurality of windows each corresponding to a different spatial region of the scene. The processor may be further configured to identify a central portion of the current frame comprising a subset of the plurality of windows. The processor may be further configured to determine a depth value of the central portion based on performing PDAF on the subset of the plurality of windows and determine a confidence value for the central portion based on the depth value and image data corresponding to the subset of the plurality of windows.