Architecture for real-time extraction of maximally stable extremal regions (MSERs) is disclosed. The architecture includes communication interface and processing circuitry that is adapted in hardware to receive a data streams of an intensity image and a depth image in real-time and provide intensity labels for image regions within the intensity image that match a given intensity threshold and provide depth labels for image regions within the depth image that match a given depth threshold. The processing circuitry is also adapted in hardware to find intensity extremal regions within the intensity image based upon the intensity labels and to find depth extremal regions within the depth image based upon the depth labels. The processing circuitry determines strong extremal regions based upon significant overlap between the intensity extremal regions and depth extremal regions. The processing circuitry then determines X-MSER ellipses parameters based upon the strong extremal regions and X-MSER criteria.