The system and method of the invention combines target image intensity into a maximum likelihood estimate (MLE) framework as in STAPLE to take advantage of both intensity-based segmentation and statistical label fusion based on atlas consensus and performance level, abbreviated iSTAPLE. The MLE framework is then solved using a modified expectation-maximization algorithm to simultaneously estimate the intensity profiles of structures of interest as well as the true segmentation and atlas performance level. The iSTAPLE greatly extends the use of atlases such that the target image need not have the same image contrast and intensity range as the atlas images.