A method and system for recognizing behavior is disclosed, the method includes: capturing at least one video stream of data on one or more subjects; extracting body skeleton data from the at least one video stream of data; computing feature extractions on the extracted body skeleton data to generate a plurality of 3 dimensional delta units for each frame of the extracted body skeleton data; generating a plurality of histogram sequences for each frame by projecting the plurality of 3 dimensional delta units for each frame to a spherical coordinate system having a plurality of spherical bins; generating an energy map for each of the plurality of histogram sequences by mapping the plurality of spherical bins versus time; applying a Histogram of Oriented Gradients (HOG) algorithm on the plurality of energy maps to generate a single column vector; and classifying the single column vector as a behavior and/or emotion.