Systems, apparatus, and methods to determine relationships, group memberships, and social networks and circles automatically through an analysis of media content, such as photos or videos, and ascertaining the connections between individuals or groups. Embodiments include identifying individuals in photos with or without other people in different situations, and developing a context based on objects in the photo. The extracted context may be stored in a predefined ontology structure in a database and analyzed to determine relationships or group memberships. A database of automatically determined relationships may be analyzed in order to provide relationship recommendations, social advice, or to sup port law enforcement investigations. The database may be continuously updated with additional information provided by individuals or obtained through a secondary socal network that includes user media or social information.