For an input image of a person, a set of object proposals are generated in the form of bounding boxes. A pose detector identifies coordinates in the image corresponding to locations on the person's body, such as the waist, head, hands, and feet of the person. A convolutional neural network receives the portions of the input image defined by the bounding boxes and generates a feature vector for each image portion. The feature vectors are input to one or more support vector machine classifiers, which generate an output representing a probability of a match with an item. The distance between the bounding box and a joint associated with the item is used to modify the probability. The modified probabilities for the support vector machine are then compared with a threshold and each other to identify the item.