A method for creating an appearance model of an object includes receiving an image of the object and creating a hierarchical appearance model of the object from the image of the object. The hierarchical appearance model has a plurality of layers, each layer including one or more nodes. Nodes in each layer contain information of the object with a corresponding level of detail. Nodes in different layers of the hierarchical appearance model correspond to different levels of detail.