Hardware architecture for real-time extraction of maximally stable extremal regions (MSERs) is disclosed. The architecture includes a communication interface and processing circuitry that are configured in hardware to receive a data stream of an intensity image in real-time and provide labels for image regions within the intensity image that match a given intensity threshold. The communication interface and processing circuitry are also configured in hardware to find extremal regions within the intensity image based upon the labels and to determine MSER ellipses parameters based upon the extremal regions and MSER criteria. In at least one embodiment, the MSER criteria include minimum and maximum MSER areas, and an acceptable growth rate value for MSER area. In another embodiment, the MSER criteria include a nested MSER tolerance value.