The present teaching relates to method, system, medium, and implementations for detecting a source of speech sound in a dialogue. A visual signal acquired from a dialogue scene is first received, where the visual signal captures a person present in the dialogue scene. A human lip associated with the person is detected from the visual signal and tracked to detect whether lip movement is observed. If lip movement is detected, a first candidate source of sound is generated corresponding to an area in the dialogue scene where the lip movement occurred.