Methods are provided for automatically analyzing and understanding activities and interactions. One method comprises receiving at least location information for one or more individual objects in a scene at a given time; applying at least one machine learning or artificial intelligence technique to automatically learn an informative representation of location trajectory data for each object; and identifying and analyzing individual and group activities in the scene based on the trajectory data. Another method comprises receiving at least location information for one or more individual objects in a scene at a given time; receiving visual information about the scene and at least one individual object in the scene; applying at least one machine learning or artificial intelligence technique to automatically learn an informative representation of location trajectory data, and to automatically learn an informative representation of visual appearance data; and combining the location trajectory and visual appearance data.