A method comprising: receiving a plurality of audio segments comprising a speech signal, wherein said audio segments represent a plurality of verbal interactions; receiving labels associated with an emotional state expressed in each of said audio segments; dividing each of said audio segments into a plurality of frames, based on a specified frame duration; extracting a plurality of acoustic features from each of said frames; computing statistics over said acoustic features with respect to sequences of frames representing phoneme boundaries in said audio segments; at a training stage, training a machine learning model on a training set comprising: said statistics associated with said audio segments, and said labels; and at an inference stage, applying said trained model to one or more target audio segments comprising a speech signal, to detect an emotional state expressed in said target audio segments.