Embodiments include selecting edgels for edge based tracking by dividing a reference image frame (RF) into N×M bins of pixels and projecting a subset of the edgels per bin into a current image frame (CF) using an estimated pose to identify valid bins of the RF as bins having their projected one edgel found within the borders of the CF. Then, K edgels of RF with different orientations from each valid bins may be selected. Then, the selected RF edgels of bins may be reduced by removing bins randomly, or first removing bins from the center of the RF of the image (then next removing the next further outward bins), until a desirable edgel number is obtained. Edge-based tracking can then be performed using the desirable edgel number, to track edges in current frame that are found in prior frame.