A method and apparatus for cross-modal matching with polarimetric thermal image data is provided. According to one embodiment, the method comprises receiving a plurality of polarimetric thermal images of a face acquired by a polarimetric thermal imaging camera, extracting features of the face from each of the plurality of images to generate a plurality of feature vectors for each of the plurality of images, compositing the feature vectors for each of the plurality of images together to form composite feature vectors and cross-modally matching the composite feature vectors with other feature vectors, in order to determine whether the face matches a face represented by the other feature vectors.