A model-based segmentation system includes a plurality of clusters (48), each cluster being formed to represent an orientation of a target to be segmented. One or more models (140) are associated with each cluster. The one or more models include an aspect associated with the orientation of the cluster, for example, the appearance of the target to be segmented. A comparison unit (124), configured in memory storage media, is configured to compare an ultra-sound image to the clusters to determine a closest matching orientation and is configured to select the one or more models based upon the cluster with the closest matching orientation. A model adaptation module (126) is configured to adapt the one or more models to the ultrasound image.