A unified framework to understand multimedia signals utilizes the loosely annotated multimedia data on the Web, analysis it in various signal domains, such as text, image, audio and combinations thereof, and builds an association graph called the “Multimedia Brain,” which basically comprises visual signals, audio signals, text phrases and the like that capture a multitude of objects, experiences and their attributes and the links among them that capture similar intent or functional and contextual relationships.